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Abstract The generalized Polya urn (GPU) models and their vari¬ 
ants have been investigated in several disciplines. However, typical 
assumptions made with respect to the GPU do not include urn mod¬ 
els with diagonal replacement matrix, which arise in several applica¬ 
tions, specihcally in clinical trials. To facilitate mathematical analyses 
of models in these applications, we introduce an adaptive randomly 
reinforced urn model that uses accruing statistical information to 
adaptively skew the urn proportion toward specihc targets. We study 
several probabilistic aspects that are important in implementing the 
urn model in practice. Specihcally, we establish the law of large num¬ 
bers and a central limit theorem for the number of sampled balls. To 
establish these results, we develop new techniques involving last exit 
times and crossing time analyses of the proportion of balls in the urn. 

To obtain precise estimates in these techniques, we establish results 
on the harmonic moments of the total number of balls in the urn. 
Finally, we describe our main results in the context an application 
to response-adaptive randomization in clinical trials. Our simulation 
experiments in this context demonstrate the ease and scope of our 
model. 


1. Introduction. A generalized Polya urn (GPU) model [ 4 ] is charac¬ 
terized by the pair (Yi,n)P2,n) of random variables representing the num¬ 
ber of balls of two colors, red and white, for instance. The process is de¬ 
scribed as follows: at time n = 0 , the process starts with (yi,052/2,0) balls. 
A ball is drawn at random. If the color is red, the ball is returned to the 
urn along with the random numbers (Dii,i,Di2,i) of red and white balls; 
otherwise, the ball is returned to the urn along with the random numbers 
(D2 i,i, D22,i) of red and white balls, respectively. Let Yi,! = yi,o + T)ii,i and 
^2,1 = 2/2,0 + -^12,1 denote the urn composition when the sampled ball is red; 
similarly, let Yi,i = 2/1,0 + L^2i,i and y2,i = 2/2,0 + ^>22,1 denote the urn com¬ 
position when the sampled ball is white. The process is repeated yielding the 
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collection {(^1,^ ^2,n); > 1 }- The quantities Ri = {{Du^n, Di2^n)','n > 1 } 

and R2 = {iD2i^n, D22,n)','n > 1 } are collections of independent and identi¬ 
cally distributed (i.i.d.) non-negative integer valued random variables, and 
i?i is assumed to be independent of i?2- We refer to 


D 


n 


^ 11 ,n D\2,n 
D21,n D22,n 


as a replacement matrix. 

In this paper, we focus on an extension of the randomly reinforced urn 
(RRU) model, a variant of the randomized Polya urn (RPU) models, whose 
replacement matrix is given by 

Dl,n 0 
0 T>2,n 

where the random variables and are supported on [0,00), rather 
than on the set of non-negative integers. Let mi := E[Di^n] and m2 '■= 
E[D2,n]- For the RRU model, a law of large numbers was established in [ 18 ]; 

i.e. 

1 ■ I{r7ii>m2} T 0 • if nil 7^ ni2. 

Zoo if mi = m2, 


(1.1) Z, = 




Tl,n + >2, 


Dn = 


Dll^n Di2^r 
D21,n D22,r 


where T|' stands for almost sure convergence and Zoo is a random variable 
supported on ( 0 , 1 ). The properties of the distribution of Zoo were studied 
in [ 1 , 2 ]. Denoting {(W,n)-^2,n); n > 1 } the number of balls of red and 
white colors sampled from the urn, one can deduce from the above LLN 
that Ni^n/n converges to the same limit as Z„. 

Notice that the limit of the RRU in ( 1 . 1 ) is always 1 or 0 when mi 7^ 
7712 . However, in applications it is common to target a specific value p G 
( 0 , 1 ). This was achieved in [ 3 ], where the modified randomly reinforced urn 
(MRRU) model was introduced. The MRRU model is an RRU model with 
two fixed thresholds 0 < /32 < Pi < 1, such that if Z„ < p2, no white balls 
are replaced in urn, while if Z„ > pi, no red balls are replaced in the urn. 
These changes occur at random times and will in general depend on mi 
and m2- Thus, even if the sequences {Diy,n > 1 } and {D2y,n > 1 } are 
i.i.d., the replacements matrices of the MRRU model are not i.i.d. Indeed 
the replacement matrix has the following representation: 


Dn = 


Dl,n ■ f{Zn-i<pi} 


0 

D 2 , 


' ^{Zn-l>P2} 
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The LLN for the MRRU when mi ^ m 2 is established as 

Oi, S. ^ ^ 

t Pi • -l-{mi>m2} ' P'2 ’ 

A second order result for Zn, namely the asymptotic distribution of after 
appropriate centering, was derived in [13]. However, the validity of the CLT 
for Ni^n/n in the MRRU model is not known. 

A critical issue in the MRRU model is that pi and p 2 are typically un¬ 
known in real applications. In this paper, we use the accruing information 
concerning the balls in the urn to provide random thresholds which converge 
a.s. to specihed targets. More specihcally, our replacement matrix becomes 


( 1 . 2 ) 


Dn = 


T>l,n • 0 

0 T>2.n • l{Z„>p2.n} _ 


where pi^n and /02,n represent the random thresholds. We call this adaptive 
urn model an adaptive randomly reinforced urn (ARRU), to distinguish it 
from the RRU and the MRRU. In this paper, we investigate the asymptotic 
properties of the ARRU model when mi 7^ m2. Specifically, we establish the 
LLN for Zn and Ni^n/n, and the CLT for Ni^n/n. Before concluding this 
section, we describe some recent works in the literature which are similar in 
spirit to the present work but are quite different from our proposed model. 

Let Hn '■= E\Dn\Hn-i], where Hn-i is the “information” up to the time 
(n — 1). This is referred to as the generating matrix. Asymptotic properties 
of the urn composition for homogeneous GPU, i.e. Hn = H for all n > 1, 
have been studied in [4] under the assumption that H is irreducible. In [21], 
the extended Polya urn (EPU) is defined as a GPU such that all the rows 
of H sum to the same positive constant, i.e. 


(1.3) 


HI = cl. 


Under the assumption that H has simple eigenvalues, second-order asymp¬ 
totic properties on the proportion of sampled color extracted from the urn 
are obtained in [21]. In [15], the limiting distribution of the proportion of 
sampled balls for homogeneous urn models are derived. In [5], weak consis¬ 
tency and asymptotic normality of the urn composition for non-homogeneous 
GPU are established. However, in [5], the sequence {Hn',n > 1} is determin¬ 
istic and converges to a matrix H satisfying (1.3). [6, 7] extend [5] to random 
generating matrices and establish almost sure convergence of the proportion 
of sampled balls. They also investigate the second-order properties. A key 
assumption in [6, 7] is (1.3). In [24] the sequence of generating matrices is 
defined as function of adaptive estimators, which guarantees the convergence 
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of Hn to a limiting matrix H satisfying (1.3). For “immigrated” urn models, 
theoretical results have been obtained in [25] under the assumptions (1.3), 
or HI < 0. These extensions do not include the RRU model, where Hn is 
diagonal, non-negative and (1.3) is not satisfied. For distributional results 
concerning large Polya urns, see [8] . We now describe application to clinical 
trial literature (see [11]). For applications to computer science, we refer the 
reader to [17]. 

1.1. Applications to clinical trials. Urn models have a long history of 
applications in clinical trials, by providing randomization procedures that 
target certain objectives (for a review, see [19]). In this context, patients are 
sequentially allocated to treatments according to the sampled colors and 
the associated responses are used to update the urn. This is referred to as 
response-adaptive, since the probability of assignment depends on informa¬ 
tion about the treatment performances. For a literature review on response- 
adaptive designs in clinical trials see [14, 16]. In an RRU model, responses to 
treatments are typically transformed by a utility function to obtain the rein¬ 
forcement values, so that the higher the reinforcement, the better the treat¬ 
ment. This yields a more ethical allocation in clinical trials, because (1.1) 
shows that the RRU assigns more patients to the superior treatment. How¬ 
ever, response-adaptive designs usually aim at obtaining good inferential 
properties by targeting a certain proportion p G (0,1), which is typically 
chosen to satisfy some optimality criteria (see [20]). For this reason, in [3] 
the RRU was modified to asymptotically attain any target allocation pro¬ 
portion, p G (0,1). This guarantees the MRRU design to have an asymptotic 
allocation within (0,1) there by incorporating ethical constraints {viz. as¬ 
signing more subjects to the superior treatment). The main issue is that pi 
and p 2 are typically functions of unknown parameters (see [20] ). The ARRU 
model presented in this paper allows pi and p 2 to be functions of such 
unknown parameters, and adaptively updates by substituting sequential es¬ 
timates for the parameters. The limiting results in this paper demonstrate 
that such procedures target the unknown optimal allocation and provide an 
appropriate randomization procedure for such trials in large samples. We 
also demonstrate by simulation that the properties hold relatively well for 
moderate sample sizes. 

1.2. Structure of the paper. The paper is organized as follows. In Sec¬ 
tion 2, we present the notation and assumptions concerning the ARRU 
model and related main results. Specifically, in Subsection 2.1, we present 
the LLN; in Subsection 2.2 we present the CLT under the assumption that 
the thresholds are updated at exponentially changing times. Subsection 2.3 
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is devoted to the implications of the main results in the context of clinical 
trials. 

In Section 3, we describe several fundamental results concerning the ARRU 
model that are needed in the proof of the CLT. Specifically, we prove that the 
harmonic moments of the total number of balls in the ARRU are uniformly 
bounded. Then, we use this to obtain a uniform Li-bound for the distance 
between the urn proportion at successive update times and the adaptive 
thresholds. In Section 4 the proofs of the main results are provided, while 
Section 5 contains results of a simulation study. Section 6 contains extensions 
to multi-color urn models. 

Finally, some remarks concerning proofs are in order. The LLN and CLT 
for Ni^n/n are deduced using the asymptotic properties of For this rea¬ 
son, in several results of this paper we will provide a detailed probabilistic 
description of the sequence n > 1}. 


2. Model assumptions, notation and main results. We begin by 
describing our model precisely. Let n> 1} and ^2 = {C 2 ,n', n> 1} 

be two sequences of i.i.d. random variables, with probability distributions 
Hi and H 2 respectively. Without loss of generality (Wlog), assume that the 
support of and ^2,n is the same. We denote it by S. Consider an urn 

containing yi^o > 0 red balls and ?/2,o > 0 white balls, and define yo = 
2/1,0 + 2/2,0- At time n = I, a ball is drawn at random from the urn and its 
color is observed. Let the random variable Xi be such that 




1 if the extracted ball is red, 

0 if the extracted ball is white. 


We assume Xi to be independent of the sequences and ^2- Note that Xi 
is a Bernoulli random variable with parameter zq = 2/1,0/2/0 ■ 

Let pifi and p 2 fl be two random variables such that pifi,p 2 ,o G (0,1) and 
Pifi > P 2 ,o a.s. Let u : S' —)• [a,6], 0 < a < 6 < 00 . If Ai = 1 and zq < pi^, 
we return the extracted ball to the urn together with Di^i = u(^iq) new 
red balls. While, if Xi = 0 and zq > p 2 fi, we return it to the urn together 
with D 2 P = u (^2,1) new white balls. If Xi = 1 and zq > pi^, or if Xi = 0 
and Zq < p 2 ft, the urn composition is not modified. To ease notation, let 
denote = l{2o</3io} ^2,0 = '^{zQ>p 2 o}- Formally, the extracted ball 

is always replaced in the urn together with 


XiDi^iWifi + (1 — Xi) D2pW2fl 

new balls of the same color to the extracted one; now, the urn composition 
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becomes 

{ ^ 1,1 = 2 / 1,0 + 

^ 2,1 = 2 / 2,0 + (1 — Xl)D2,lW2fi. 

Set Yi = yi,i+l2,i and Zi = Yi^i/Yi. Now, by iterating the above procedure 
we define pi^\ and p2,i to be two random variables, measurable with respect 
to the (T-algebra Fi = + (1 - with pip^hp ^ (0,1) 

and Pip > P2P a.s. Let mi = J u{y) pi {dy) and m2 = j u{y) p2 (dy) be 
the means of > 1 } and {-D2,n;ii > 1 }, respectively. We assume 

throughout the paper the following condition: 

(2.1) mi / m2. 

The urn process is then repeated for all n > 1 . Let pi^n and p2,n be two 
random variables, measurable with respect to the u-algebra 

Xn = cr {Xi,Xi^ip + (1 — Xi) ^2,1, ■■■,Xn,Xi^i^n + (1 “ -^n) C 2 ,n) , 

with pi^n,P2,n G ( 0 , 1 ) and pi^n > P2,n a.s. We will refer to pj^n J = 1,2 as 
threshold parameters. At time n + 1, a ball is extracted and let = 1 

if the ball is red and = 0 otherwise. Then, the ball is returned to the 

urn together with 

Xn+lDi^n+l^l,n T (1 A^n+l) D 2 ^n+l^ 2 ,n 

balls of the same color, where Dgn+i = ii(?i,n+i), L>2,n+i = u{i2,n+i), 
bLi,n = 1{Z„<P1,„}, bL2,n = l{z„>p2,n} Zn+i = Li,n/L"n for any n > 1. 
Formally, 

r Yi , n+l = 2/1,0 + Er=l' 

I b2,n+l = 2 /2,0 + (1 - Xi) D 2 pW 2 ,^-l 

and Yn+i = + Y2,n+i- If Xn+i = 1 and Zn > pi,n, i-e. hFgn = 0, or 

if Xn+i = 0 and Zn < P2,n, i-e. lF2,n = 0, the urn composition does not 
change at time n + 1 . Note that condition pi^n > P2,n a.s., which implies 
II^i,n + IL2,n > 1, ensures that the urn composition can change with positive 
probability for any n > 1, since the replacement matrix (1.2) is never a 
zero matrix. Since, conditionally to the cr-algebra Xn, Xn+i is assumed to 
be independent of ^1,^2, Xn+i is conditionally Bernoulli distributed with 
parameter Zn- 

We will denote by Ni^n and A^2,n the number of red and white sampled 
balls, respectively, after the hrst n draws, that is = X)r=i Xi and A'"2,n = 
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Yll=i (1 “ ^i)- Let Pi and p2 be two constants such that 0 < /92 < /Oi < 1- 
We will adopt the following notation: 

Pn ■ Pl,nl{mi>m 2 } P2,nl{mi<m2}) 

P • Pll{mi>m2} P2l{mi<m2} ■ 

2 . 1 . Law of large numbers. Our first result is concerned with the LLN. 


Theorem 2.1. Under assumptions (2.1) and 


(2.2) 

lim Pn 

n^oo 

we have that 


( 2 . 3 ) 

lim Z„ 

n^oo 


From Theorem 2.1 we can obtain the convergence of sampled balls, namely 

Ni,n/n. 


Corollary 2 . 1 . 
( 2 . 4 ) 


Under assumptions ( 2 . 1 ) and ( 2 . 2 ), 


V 

hm - 

n—^cxD n 


= p a.s. 


2.2. Central limit theorem. We next study the limit distribution of pro¬ 
portion of sampled balls By the description of the model, depends 
on the sequence pj^n, j = 1 ) 2 . However, frequent changes to pj^n make the 

sequence more erratic. To stabilize the behavior of > l|, we 

hx a constant q > 1 and introduce the sequence pj^n J = 1) 2 as 

( 2 . 5 ) Pj^n ■■= Pj,[gi], as [g*] <n< [g*+^] , 

for i G N; that is, we adapt the threshold parameters to change “slowly” 
at exponential times | [g*] , i = 1, 2, ..|. An alternative definition of pj^n j = 

1,2, which is used in some proofs, is the following 

( 2 - 6 ) {pi,ni P2,n) •= [^agq{n)], 

for any n > 1 . We will denote by 


Pn Pl,nl{mi>m 2 } P2,nl{mi<m 2 }■ 

We now turn to the statement of the CLT. In the following represents 
the convergence in distribution. 
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Theorem 2.2. Let pi^n and p2,n be as in (2.5). Assume that for any 
e > 0 and j = 1,2, there exists 0 < ci < oo such that 

(2.7) P {\ pj ^ n - Pj\ > e) < ciexp(-ne^), 
for large n. Then, under assumption (2.1), we have that 

(2.8) ^/n(J^^-pn^ A M {0,p{l-p)). 


where pn = • 

Remark 2.1. The result of Theorem 2.2 continues to hold if (2.7) is 
not satisfied, but (2.2) and the following conditions hold: 

(cl) limsup.,.,^^^/nE[\pn - p\] < oo. 

(c 2 ) There exists e G ( 0 , 1/2) such that pj^n S 1 “ e] O'-S- for any n > 1 , 

/ = 1 , 2 . 

Theorem 2.2 introduces an asymptotic bias for A^i „/re given by {pn — p). 
We show that this bias is exactly of order 0 (n“^/^); our next proposition 
makes this observation precise. 

Proposition 2.1. Let pi^n ond P2,n be as in (2.5). Then, under assump¬ 
tion (2.1) and (2.7), 

( 2 . 9 ) limsup n ■ E [\pn — pI"^] < oo. 

n^oQ 

Remark 2.2. The result of Proposition 2.1 continues to hold if (2.7) is 
not satisfied, but the following condition holds: 

(c 4 ) \imsupn^,^nE[\pn-p\^] < oo. 

In the case when pi^n = Pi and p2,n = P2 for any n > 0 , Theorem 2.2 
provides a CLT for the allocation proportion of MRRU model. This is sum¬ 
marized in the following corollary: 


Corollary 2.2. In a MRRU, under assumption (2.1), we have that 





CLT FOR AN ADAPTIVE URN MODEL 


9 


2.3. Application to clinical trials (revisited). Consider two competing 
treatments 7i and T). The random variables and ^2,n are interpreted 
as the potential responses to treatments 71 and Ti, respectively, given by 
subjects that sequentially enter the trial. At all times n > 1, a subject is 
allocated to a treatment according to the color of the sampled ball and a 
new response is collected. Note that only one response is observable from 
every subject, that is Xn^i^n + (1 ~ Xn) (, 2 ,n- The function u transforms the 
responses into reinforcements Di^n and T*2,n that update the urn. Typically, 
u is chosen such that 71 (or 71) is considered the superior treatment when 
mi > 1712 {^1 < We assume there exists a unique superior treatment, 
which is formally stated in assumption (2.1). 

We now describe the role of the sequences {/5i^„;n > 1} and {p 2 ,n',n > 
1} in clinical trails. Assume the distributions pi and p 2 are parametric, 
depending on the vectors 6 i and 62 respectively, with 0 = { 61 , 62 ) G 0 C 
R^, with d > 1. Let dn = (^ 6 i n, 62 ^n^ be an estimator of 6 after the first n 

allocations, so that dn is measurable with respect to the a-algebra Rn- We 
assume that the distributions pi and p ,2 are parametrically independent, in 
the sense that pi does not depend on 62 and fi2 does not depend on 61. 
Hence, 6i^n is computed with the Ni^n observations : Xi = l,i < n}, 
while 02 ,n is computed with the N 2 ,n observations {^2,1 : Xi = 0,i < n}. 
Thus, {piy,n > 1} and {p 2 ,n',n > 1} are dehned as follows: 

(2.10) Pi^n ■= fl and P2,n ■■= f2 (^2,n) , Vn > 1, 

where /i : 0 —)• (0,1) and /2 : 0 —)• (0,1) are two continuous functions such 
that 

fl ( x ) > f 2 ( x ) , V® € 0; 

this implies pi^n > P 2 ,n a.s. for every n > 1. Moreover, set 
Pi := fl (0) and p 2 := /2 {6) ■ 

The LLN presented in Theorem 2.1 suggests a direct interpretation for 
the functions fi and /2 in a clinical trial context: fi {6) and /2 {6) represent 
the desired limiting allocations for the sequence Ni^n/n, in case the superior 
treatment is 7i (mi > m2) or T 2 (mi < m2), respectively. This is a great 
improvement, since the design can target an arbitrary known function of all 
the parameters of the response distributions. 

Ideally, fi and /2 are chosen to obtain good statistical properties from 
the design. Typically, in clinical trials, a design is constructed to satisfy 
certain optimality criteria related to its statistical performances (e.g., power; 


10 


GHIGLIETTI ET AL. 


see [20]). Letting r/ (0) denote the limit proportion of subjects to be allocated 
to treatment 71, this design can be obtained by the urn model described in 
Section 2 by choosing /i (6) = f 2 (0) = r] (0). However, in some experiments, 
ethical aspects are important and the main goal may be to assign fewer 
subjects to the inferior treatment; in this case we choose fi (0) ~ 1 and 
/2 (0) ~ 0. Designs requiring both ethical and statistical goals can also be 
obtained from our design, by setting fi (0) > r] (0) > f 2 iO). For instance, 
we may take 

(2.11) /i (0) =p-r/(0) + (l-p)-l, /2 (0) =p-r/(0) + (l-p)-0, p £ (0,1] , 

where p is a biasing term, which introduces a trade-off between the ethics 
and statistical properties. 

Finally, it is worth emphasizing that conditions (2.2) and (2.7) required 
in the LLN of Theorem 2.1 and in the CLT of Theorem 2.2, respectively, 
are straightforwardly satished when we take On to be maximum likelihood 
estimators (MLEs) for 0. 

Moreover, condition (c2) in Remark 2.1 is equivalent of the assumption 
that the ranges of fi and /2 are subsets of [e, 1 — e], for some e G (0,1/2). 

3. Harmonic moments and related asymptotics. 


3.1. Harmonic moments. In this subsection, we show that the harmonic 
moments of the total number of balls in the urn are uniformly bounded. 
This is a key result which is needed in several probabilistic estimates, and in 
particular in the proof of the CLT. More specifically, as explained previously 
the results concerning the asymptotic behavior of Ni^n/n, depend critically 
on the behavior of {Zn—pn)- In Subsection 3.2 we provide bounds for Yn{Zn— 
Pn), by using comparison arguments with the MRRU model. Now, to replace 
the random scaling Yn by the deterministic scaling n, one needs to investigate 
the behavior of njYn- Our next theorem provides a precise estimates of the 
jth j] 2 oment of njYn for any j > 0. 


Theorem 3.1. 
have that 


Under assumption (2.1) and (2.7), for any j > 0, we 


sup E 

n—>oo 



< OO. 


In the proof of Theorem 3.1, we need the following lemma that provides 
an upper bound on the increments of the urn process by imposing a 
condition on the total number of balls in the urn Yn- Hence, the proof of 
Theorem 3.1 is reported after the following result. 
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Lemma 3.1. For any e € (0, 1), we have that 
(3.1) c {\Zn+i-Zn\<e}. 


Proof. The difference (Zn+i — Zn) can be expresses as follows: 

hl,n T ^n+l^^l,nDl,n+l Pi,n 

Yn + ^n+lWi^nDl ,n+l + (1 — Xn+i)W2,nD2,n+l Yn 

Consider {Zn+i > Zn}, since the case {Zn+i < Zn} is analogous. Note 
that {Zn+i > Zn} implies that {Xn+i = 1} and {bTi,n = !}• Then, since 
Di^n+i < b a.s., on the set {Zn+i > Zn} we have 


Zn+l Zn 


< 


Ll,n T T^l,n+1 Yl^r, 
Yn T Y)i n-\-l 
h^l,n+l 


Yn 


Dl^n+1 T Yn 


{l-Zn) < 


b + Yn 


< e, 


where the last inequality follows from {Yn > 6(1 — e)/e} in (3.1). □ 


Proof of Theorem 3.1. In this proof, when we have set of integers 
{[ai],.., [6i]} with ai,6i ^ N, to ease notation we will just write {ai,..,6i}, 
omitting the symbol [■]. First, note that, since Di^i,D 2 ^i > a a.s. for any 
i > 1 and Tq > 0, we have that 

n 

To + + Zl2,Ul - X,) 1F2 ,*-i) 

i=l 

n 

To + a • + (1 - X,) VF2 ,*-i) , 

i=l 
n 

Yo + a - Y, {XiWi,i_i + (1 - Xi) W 2 ,i-i ), 

i=n(3 

for any /3 G (0,1). To keep calculation transparent we choose /? = 1/2. 
We recall that, by construction, we have that ITi,,-!, lF 2 ,i_i G {0; 1} and 
+ W 2 ,i-i > 1 for any i > 1; hence, the random variables WkFi,i-i + 
(1 — Xi) W 2 ,i-i are, conditionally to the fi-algebra Fi-i, Bernoulli distributed 
with parameter greater than or equal to min{Zj_i; 1 — Zi-i}. Hence, the be¬ 
havior of Yn is intrinsically related to the behavior of Zn- 


(3.2) 


Pn = 


> 


> 
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Thus, let us introduce the sets (down), (center) and (up) as 
follows: 


Ad,r. 


U {^*<4 

nl2<i<n 


A 


c,n 


n c ]}\, 

nl2<i<n 


Au,n := ■( IJ {Zi> l-c}\ , 

nl2<i<n J 

where c G (0,1) will be appropriately fixed more ahead in the proof. Then, we 
perform the following decomposition on the behavior of {Z^; n/2 < i < n}. 


E 


n 


< (^) ■P{Ad,n)+E 




n 


+ { 


On the set ^c,n the process {Zj; n/2 < i < n} is bounded away from the ex¬ 
treme values {0; 1}; Hence we can use comparison arguments with a sequence 
of i.i.d. Bernoulli random variables with parameter c to get the boundedness 
of E {njYny lyic,™ ■ After that, we will focus on proving that P{Ad^n) and 
P{Au,n) converge to zero exponentially fast. 

First, note that on the set Ac n the random variables 


+ (1 - Xi) 


are, conditionally to the u-algebra Bernoulli with parameter with 

parameter greater than or equal to c for any i = n/2, ..,n. Hence, if we 
introduce {Bi]i > 1} a sequence of i.i.d. Bernoulli random variable with 
parameter c, from (3.2) we have that 


E 



1 

< —E 
(P 


n 


ho/« + Er=n/2^* 


j 


We now show that 


limsup E 

n—)-oo 


n 


Kc 


+ Eti 


j' 


< oo. 


with Kq = Yoja. To this end, we apply Theorem 2.1 of [12], with no = 1, 
p = j, Zi^n = Bi + Yq/u for i < n. All the assumptions of the theorem 
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are satisfied in our case. In fact, at first we have E 


y-v 


< oo because 


E 


{Yo + B^) 


-3 


< Kq ^ < oo. 


lat Zi^n are identically distributed for all i < n, since Bi are 


Secondly, note t 

i.i.d. Bernoulli of parameter c. Finally, Zn converges in distribution, since 
Zn = Bi/n + Kq c + Kq. Heuce, by Theorem 2.1 of [12], it follows 


that E 


Z, 


-p 


is uniformly integrable. As a consequence. 


limsup E 

n^oo 


n 


Ko + T.UB^ 


limsup E < oo. 

n—)-oo 


Now, we will prove that P^Ad^n) and P{Au^n) converge to zero exponen¬ 
tially fast. We will show that this occurs because pi^n and p 2 ^n are bounded 
away from the extreme values {0; 1}, with probability that converge to one 
exponentially fast. Formally, fix e > 0, such that pi -|- e < 1 and p 2 — e > 0, 
and define := n“, a G (0,1), for any n > 1. Now, for any n > 1 define 
the following sets: 


A\^n 

A2,n 

A3,n 


sup {pi,i} > pi + e } , 

i>an 


= \ .inf {P2,i} < P2 - e , 

2 ^ Oin 


\ inf {minjl - pi^p, p2,i}} > minjl - pi; P2} - e L 
f l>an J 


where we recall that pi,i and p2,i are the adaptive thresholds. Note that 
Ai^n U A2,n U A^^n = We have that 

P{-^d,n) Y P [A\^n) + P {A2,n) + 2-* (^3,^, H Arf , 

P{Au,n) Y P (^Ai^n) + P {A2,n) + P {As^n Au^n) ■ 


First, we prove that P and P (A2,n) converge to zero exponentially 

fast. Consider the term P {Ai^n). From the definition of we obtain 


P{Ai^n)=P 


U {pi,i > Pi + e} 


i i>(y.n 


B {pi,i > Pi + e). 

i>a„ 


From (2.7), for large i we have that 


P {pi,i > Pi + e) < Cl exp {-ie^) , 
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with 0 < Cl < oo. Hence, using the fact that Yn is increasing we have that 
P{M,n) < P {h,i > /^i + e) 

i>a„ 

< Cl exp (— 
i>an 

= Cl exp (-anC^) • 

Similar arguments can be applied to prove P (^2,n) —^ 0 exponentially fast. 

Finally, we show that P {A^^n H and P {A^^n H Au^n) converge to 

zero exponentially fast. Consider P {A^^n C ^d,n), since the proof for 
P {A^^ri n Au^n) is analogous. First, let introduce 4> := min{/92; 1 — Pi}, and 
rewrite A'^^n as follows: 

A^^n = \ inf {P2,n; 1 - h,n] > 0 “ £ 

Dehne the set A^^n as follows: 

Ad,n := I Pi {Zi < cj 
I OLn<i<nl1 

We now set an appropriate value of c such that 

(3-3) {^ISjn C Ad^n} C ■|^j 43 n , 

for any n > 1. To do that, we need to set c such that {Zi > cj C {Zi^i > c} 
for any i > a„. First, note that on the set ^3,n, {P 2 ,i > {4> — e)} for any 
i > an- Hence, for any c < {(j) — e), if {c < Zi < {(j) — e)} we have W 2 ,i = 0, 
that implies ZiJ^i > Zi and so Zj+i > c. Alternatively, if {Zi > [cp — e) > cj, 
the set {ZiJ^i < Zi} is possible, and hence we have to bound the increments 
of Zn to guarantee that ZiJ^i > c, i.e. set c such that 

|Zj+i - Zi\ < {(p - e) - c, Vi > 0. 

Using (3.1), we obtain 

(3-4) c<po:= TT^ •(</>-£)■ 

ro + 0 




This guarantees (3.3) holds for any n > 1. 
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We now show that P{A 3 ^n H ^d,n) converges to zero exponentially fast. 
To this end, first note that on the set we have p 2 ,i > /O2 — e for any 

i = Q!n,..,n/2; moreover, on the set Ad^n, we have Zj < pQ for any i = 
an,--,nj2. These considerations imply that W 2 ,i = 0 and Wi^i = 1 for any 
i = On, ■■,nj2, on the set A^^n H Ad^n- Hence, we can write 


> 1 / 1,0 


(3.5) 


Zn/2 — 


Ya. + EriL (yo + anb) + a ZtL 


where the inequality is because ^ 2/i,o, Ya„ < yo + anb and Di^i > a 

a.s. for any i > 1. Now, define for any n > 1 the set ^4^^ as follows: 


■^4,n • — 



PO 


a(l -po) 


{yo + anb) 


and consider the set A^^n hi Ad^n h ^4,n- On the set A^^n O Ad^n we can use 
the definition of A^^n in (3-5), obtaining 

n Ad^n O ^4,nj’ C ^ Po} O • 

However, {Z„/2 > Po} O Ad^n = 0- Hence, P{A^^n h Ad^n O ^4,n) = 0 and it 
is sufficient to show that P n Ad^n O A'^n^ converges to zero exponen¬ 
tially fast. 

To this end, note that on the set A^^n O Ad^n we have Zj+i > Zi for 
any i = ..,n/2, since we previously showed that W 2 d = 0 and Wi^i = 1. 

Hence, on the set A^^n^^d,n-, {Xi,i = an -,.., n/2} are conditionally Bernoulli 
with parameter pi > Za^ a.s. Now, let denote with {gi^n, * = 1,.., u/2 — an} 
a sequence of i.i.d. Bernoulli random variable with parameter zo^n, defined 
as 


^0,n ■ — 


1/1,0 


< Za 

- 0:73 


a.s.; 


yo + anb 

it follows that P (^A^^n h Ad^n h is less than or equal than 


(3.6) 



Po 


a(l -po) 


(yo + anb) 


Finally, we use the following Chernoff’s upper bound for i.i.d. random vari¬ 
ables in [0,1] (see [10]) 


(3.7) 


P {Sn < Co • T/[S'n]) < exp 


2 
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with Co € (0,1) and Sn = In our case, we have that (3.6) can be 

written as P {Sn < c„ • E[Sn]), where Sn = Qi,n and 



i/1,0 

(yo + anb) 


and Cn 


Po {yo + aubf 

a(l -po) yi,o(n/2 - a„)’ 


since c^ —>■ 0 , we can dehne an integer no such that c„ < cq for any n > no, 
so that 

P {Sn < Cn ■ E[Sn]) < P {Sn < Cq ■ E[Sn]) ■ 

Hence, by using (3.7), for any n > no we have that 


P {Ao^n n 


< 



(1 - Co)" 


E[Sn 


which converges to zero exponentially fast since 


E[Sn] 


yi,o(n/2 - an) n 

- r\j - 

yo “ 1 “ anb an 


This concludes the proof. 


n 


1 — 0 


□ 


Remark 3.1. The result of Theorem 3.1 can he also obtained relaxing 
assumption (2.7). In that case, we need condition (2.2) and (c2) to be sat¬ 
isfied. Then, the proof is the same by setting 

Ai^n ■= \ sup{pi,j} > 1 - e 
I i>an 

A2,n ■= \ inf {P2,i] < e 

i>an 

A^^n := \ inf {min{l - piAP2,i]] > e 

l>Otn 

where 0 < e < 1/2 is such that pj^n £ [C) 1 “ c] for any n > 1 and j = 1 , 2 . 
Then, P{Ai^n) = P{A 2 ,n) = 0 for any n > 1. 

3.2. A uniform bound. In this subsection, we provide a uniform bound 
for the scaled difference between Zt and pt (which is J-)--measurable). To 
make precise statements, we start by defining some notations. Set Aj^k ■= 
sign{mi-m 2 ) {pqj+k - and fj^k '■= for any j > 1 and any 

k = 1, ..,dj, where dj := — Note that, since from (2.5) Pi^gj^k — Pi,q 3 
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for any k E { 1 , dj}, we can also write Aj^k = sign{mi — m 2 ) {pqj — ■ 

Let {tj-; j > 1} be a sequence of stopping times defined as follows: 

(3.8) 


r,- : = 



k>l : G [-b, 0 ] } 


if { A:> 1 : G [-6,0] } / 0; 

otherwise. 


In Theorem 3.2 we provide a Li-uniform bound for the scaled distance 
among urn proportion Zqj_^^ and the threshold Pqj^k: 011 the set {tj < k}. 


Theorem 3.2. Let pi^n and p 2 ,n be as in (2.5). Then, under assump¬ 
tion ( 2 . 1 ) and (2.7), there exists a constant C > 0 such that 


(3.9) 


sup sup E 

j>l l<k<dj 


• \^j,k\^{Tj<k} 


< c , 


where dj = . 


The proof uses comparison arguments with the MRRU model and re¬ 
lated asymptotic results. Hence, we first present the results concerning the 
MRRU model in Subsection 3.2.1. The proof of Theorem 3.2 is reported in 
Subsection 3.2.2. 


3.2.1. Estimates for the MRRU model. In this subsection, we present 
some probabilistic estimates concerning the MRRU model which are needed 
in the proof of Theorem 3.2. We recall that for the MRRU the threshold 
are fixed, i.e. pj^n = Pj for any n > 1, j = 1,2. Hence, in this subsection 
we consider lUi^„ = l{z„<pi} and lT 2 ,n = 1 {Z„>P 2 }- start by introducing 
some quantities related to the MRRU model. Let {T„; n > 0} be the process 
defined as 

(3.10) Tn := sign{mi - m 2 ) -Ynip- Zn ), 
which is sometimes useful to represent it as follows: 

Tn = sign{mi - m 2 ) ■ {pY 2 ,n - (1 - p)Yi^n) ■ 

Then, let to be the following stopping time 

(3.11) to := fof { k >0 : Tfc G [—6,0] } . 

Let 

Sn ■= {0 <k <n: Tn-k G [-6,0]} , 
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and let {sn',n > 1} be a sequence of random times defined as 

/mf{5„} if5„/0; 

(3.12) Sn= < . 

I oo otherwise. 

where we recall that b is the maximum value of the urn reinforcements, i.e. 
Di^n, D 2 ^n < b a.s. for any n > 1. Note that by definition {s„ = oo} = {to > 
nj. In Theorem 3.3 we provide the L2-uniform bound for Yn{Zn — p), on the 
set {to < nj. 


Theorem 3.3. For an MRRU, under assumption (2.1), there exists a 
eonstant C > 0 such that 


(3.13) 


supE 

n>l 


(y„|p-z„l)2 


to < n 


< C. 


The proof uses the boundedness of the moments of the excursion times 
Sn, which is provided in Theorem 3.4. Hence, we first present Theorem 3.4 
and then we report the proof of Theorem 3.3. 


Theorem 3.4. For an MRRU, under assumption (2.1), there exists a 
eonstant C > 0 such that 

sup{£; [4|io < n]} <C. 

n>l 

In the proof of Theorem 3.4, we need to couple the MRRU model with a 
particular urn model {Zn;n > 1}. The processes are coupled, in the sense 
that: (i) the potential reinforcements are the same, i.e. = F)i^n and 

D 2 ,n = F) 2 ^n a.s.; (ii) the drawing process is defined on the same probability 
space, i.e. Un = Un a.s. where {[/„; n > 1} and {Un] n > 1} are i.i.d. uniform 
random variables such that Xn+i '■= l{!7„+i<z„} and X^+i := l|fj ^^^2^|for 
any n > 1, respectively. 

We now describe the urn model {Zn;n > 1}. Fix a constant yo G (0,1})] 


and zq 

= pi- The 

process {Z^, 

n > 1} 

evolves as follows: 

if Sn 

_i = 0, i.e. 

Zn-l 

> pi, then X 


and 



' Yl,n 

= Pi 

■yo + 



(3.14) 

< 








Y2,n 

\ 

= (1 

-Pl) • 

o 

+ 

1 

D2, 
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if Sn-l 

(3.15) 


A: > 1, i.e. Zn-i < pi, then and 

^l,n — ^l,n—1 “1“ 

< 


Y2,n — Y2,n-1 + ( f “ 


where Yn := Yi^n Y h2,n and Zn := Yi^njYn- The urn model is well defined 
since Sn-i is J-'n-i-measurable. It is worth noticing that Zn represents a 
Generalized Polya urn evaluated after exactly (sn-i + 1) steps, with initial 
composition pipo red and pi (1 — yo) white balls. 


In the next lemma, we state an important relation among the MRRU 
model and the process {Z^, n > 1}, needed in the proof of Theorem 3.4. 


Lemma 3.2. Consider the urn model {Zn',n > 1} defined in (3.14) 
and (3.15) coupled with the MRRU process {Zn]n > 1}. LetTn ■= sign(mi — 
m 2 ) ■ Yn(^p — Zn^ for any n > 1. Then, on the set {3j < n : Tj < 0}, we 
have that 

{Tn > 0} C {f; > T„| . 

Proof. Wlog assume mi > m2, which implies p = pi and T„ = Yn (pi — Zn)- 
Sometimes, we will prefer the following expression of Tn 

fn = Plp2,n - (1 - Pl)hl,n- 

The proof will be by induction. Note that, on the set {3j < n : Tj < 0}, is 
almost surely finite. On the set {sn = 0}, i.e. {Tn < 0}, we can immediately 
show that {T„+i > 0} implies {T„+i > Tn+i} and {Zn+i < Zn+i}- In fact, 
from {Tn < 0} and {T„+i > 0} we have Xn+i = 0 and IT2,n = 1, so that 

Tn+l — Tn + piD2^n+l Y piD2^n+l — PlD2,n+l — Yn-\-l 


and 


Zn+l — 


ZnYn 


> 


Pi 2/0 


- -=- 

Yn + D2,n+1 yo + T2,n+1 


— Zn+l- 


Now, consider the set {s„ > 1} and assume by induction hypothesis that 
(3.16)^ 

I Tj > Tj > 0, Zj < Zj < pi, V / = n - + 1,..., n | 0 {T„+i > 0} . 
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Then, we will show that T„_|_i > T„_|_i and Zn+i < Zn+i- Since T„ = piY2^n — 
(1 — / 9 i)Yi^n, we note that 

n+1 

Tn+l = Tn-sn + ^ [Pl (1 “ ^i) D2,iW2,i-l — (1 — Pi) XiDi^iWi^i-i] , 

i=n—Sn + l 

where we recall that for the MRRU model Wi^i = l{z.<pj} and W2^i = 
Since Tn-s„ < 0, lT2,i-i < 1, by (3.16) = 1 for any i = n- 

Sn + 1 , n, = 0 , and by construction Di^i = Di^i and £>2,1 = T>2,i, 

we have 


fn+l < 


n+l 

E 

i=n—s„+l 


Pi (1 — Yi) £>2,1 


(1 — pi) XiDi^i 


Moreover, by (3.16) we have X^+i = l{Ui+i<Zi} > ^{u,+T_<Zi} = 
any i = n — + 1,..., n. Hence, we can write 

n+l n+l 

7n+l < Pi ^ (^1 — Xi^ £>2,i — (1 — Pi) ^ XiDl,i = ^n+l- 

i=n—Sn+l i=n—Sn+l 


Similarly, we can prove that < Z^+i- Note that 
Zn-sJ^n-s^ + T.tLs ^+1 


Zn+1 — 


+ Er=n-.„+l + Er=n-..+l (1 - X£) D2,^W2,^-l 


n+l 


Now, since ^n-s„ > pi, Yn-Sn > Vo and Xi+i > Xj+i for any i = n - Sn + 
l,...,n, it follows that 


plyo + u=Ls^+lXiD^, 


Zn+l Y 


m + Er=n-s„+l X^Dl,^ + Er=n-.„+l [1-XAD2 


n+l 


= 


n+l) 


which concludes our proof by induction. 


□ 


Proof of Theorem 3.4. Wlog assume mi > m2, which implies p = pi 
and Tn = Yn{pi — Zn)- The structure of the proof is the following. The 
aim is to show that P (s„ = k\to < n) converges to zero fast enough such 
that E [s^|to E n] is bounded. To this end, we consider the urn model 
{Zn',n > 1} defined in (3.14) and (3.15) coupled with the MRRU model, 
such that P (sn = k\tQ < n) can be expressed in terms of {Zn‘, n > 1}. After 
some calculations, this is provided by Lemma 3.2. Moreover, we compare 
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{Z„;n > 1} with a Generalized Polya urn model, whose moments are uni¬ 
formly bounded. 

First, for any re > 1 note that 

n 

E[sl\tQ<n] = 'Y^k^P{sn = k\tQ<n), 

k=l 


since P (s„ = oo|to <n) = P (to > n\tQ < re) = 0. In fact, by definition 

f'O — ^ U.S. 

Before considering the urn model re > 1}, we express P (sn = k\to < re) 
in terms of {T„; re > 1}. Note that in the MRRU, if Tj > —b for some j > 0, 
then P {Tn < —b) = 0 for any re > j. In fact, when T„ > 0 {Z^ < pi) 
we have T„+i > —6, because the reinforcements are bounded by b and 
so \Tn+i — Tn\ < b a.s.; while when —b<Tn<0 {Z^ > pi) we have 
Tn+i > Tn > —b, because Zn > pi implies IFi,n = 0 and so the urn is not 
reinforced by red balls, i.e. T„+i > T„. As a consequence, since Tto > -b by 
dehnition, on the set {re > to}, we have {Tn ^ [—^,0]} C {T„ > 0}. Hence, 
since to < n — Sn, we have for all 1 < k < n 

(3.17) 

P {sn = k\to < re) = P {Tn-i > 0} n {Tn-k < 0} |to < re^ 

< p (n^Jo {^n-i > 0} I {Tn-k < 0} n (to < re}) 

< P {Tn-i > 0} iTn-k < o) , 

where the last inequality follows from {Tn-k < 0} C {to < re}. To deal 
with (3.17), we consider the urn model {Zn-, re > 1} defined in (3.14) and (3.15). 
From Lemma 3.2, we have that, on the set {3j < n : Tj < 0}, the event 
{Tn > 0} implies {T„ > T„}. Hence, we have that 

(3.18) 

P (n-To > 0} \Tn-k < o) <P {fn-i > o} \Tn-k < o) 

= p(nii{zf <pi}), 

by construction, where {Z^; i > 1} is the proportion of red balls of a Gen¬ 
eralized Polya urn, starting with a proportion of Zg = pi and an initial 
number of balls = yg, and the same reinforcements distributions as 
and P2,n- 

Now, let be the first time the process Zf is above pi, i.e. 


gG f inf { * > 1 : Zf >pi] 
1 oo 


if { i > 1 : Zf > pi } / 0; 
otherwise. 
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It can be shown using standard arguments that there exists feo € N such 
that for any k > ko, there exist 0 < ci,C2 < oo 


P = k) < Cl exp {—C2k) , 


which implies that E [exp( 7 s^)] < oo for some 7 > 0. 
Now, returning to (3.18), we have that 


p[r^U{z?<Pl ]) 

Thus we have for any k>l 

P{Sn 


and hence 

E [ 4 |io < n] 


This concludes the proof. 


E 


= P{s^ >k) < 


{soy 


k^ 


= k\to < n) < 


k^' 


'^k^P{Sn = k\to < 


n) 


k=l 


< 


< C < 00 . 


k=l 


9± 

k^' 


□ 


Proof of Theorem 3.3. Wlog assume mi > m 2 , which implies p = pi. 
Since T„ = Yn {pi — Z^) we want to prove 

supn>i E [T^\to < n] < 00 . 

Let Sn be the random time dehned in (3.12). Then, since |Tj_|_i —Ti\ <b a.s. 
for any i>l and from (3.12) £ [~^)0], we have 


E [7 1*0 < ^ E ^ ^ *4 {*0 < n}] P (s„ ^ / 1*0 < n) 


+ E® 


1=1 


1=0 

n—l 


{Ti +1 — Ti) + Tn-l j — 1 } 


\i=n—l 


P{Sn = l |to < n) 


< Y,{l + lfb^P{Sn = l\to<n). 

1=0 

Now, using [I + 1)^ < 4/^, we have that 

n 

^{1 + 1)^ b‘^P {sn = l\to <n) < Ab"^ ■ E [s^|to < n] . 


1=0 
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Finally, using Theorem 3.4 we have that the last quantity is uniformly 
bounded by a constant C independent of n, so the proof is concluded. □ 

Remark 3.2. From the proof of Theorem 3-4, we have that the con¬ 
stant C is independent of the initial proportion Zq. Moreover, C provides a 
uniform bound for any other MRRU with initial number of balls > Yq. 


3.2.2. Proof of Theorem 3.2. 


Proof. Wlog, assume mi > m 2 , which implies p = pi. First, fix j S N 
and apply Cauchy-Schwarz, so obtaining 



\ 2 



f q> \ ^ 


VI 


E 

(u) 


Since E 
that 



is uniformly bounded by Theorem 3.1, it remains to prove 


E 





< C, 


for any j > 1 and any k = 1, ..,dj. To this end, fix j S N and note that since 
Pi,qi-i-k = Pi,qi for any k G {l,..,dj}, the process {Zqjj^i,-,k = l,..,dj} can 
be considered as the urn proportion of the MRRU model, with initial com¬ 
position {Yi^qj ,Y 2 ^qj) and fixed threshold parameters pi qj and P 2 ,qj- Then, 
for each j G N we can apply Theorem 3.3, with to defined in (3.11) equal to 
Tj, SO obtaining 


(3.19) 


E 





< Cj, 


where Cj is a constant depending on the initial composition (Ti,qj ,^ 2 , 9 ^ )• 
However, from Remark 3.2 we have that there exists a uniform bound C > 0 
such that Cj < C for any j > 1, since all the processes {Zqj_^_j^, k = 1, .., dj} 
j > 1 can be considered as MRRU with initial number of balls > Tq; this 
concludes the proof. □ 


4. Proofs of the main results. Here, we present the proofs of the re¬ 
sults described in Section 2. Subsection 4.1 is dedicated to the proof of The¬ 
orem 2.1 (LLN) and the related preliminary results. Then, in subsection 4.2 
we report the proof of Theorem 2.2 (CLT) together with Theorem 4.1, a 
new result needed to compute that proof. In the last subsections, the proofs 
of the remaining results of Section 2 are gathered. 
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4 . 1 . Proof of the LLN. We start by reporting some preliminary results 
needed in the proof of the LLN. Initially, we show that the number of balls 
sampled from the urn Ni^n, A^2,n and the total number of balls in the urn Y^, 
increase to infinity almost surely. To do that, we first need to show a lower 
bound for the increments of the process Yn, which is given by the following: 


Lemma 4 . 1 . For any i>l, we have that 


E[Yi-Yi_^\Pi_^] 


V 2/0 + (* -1) & y 


Proof. First, note that 

Since Xi and Di^i are conditionally independent with respect to Fi-i, and 
Wi^i-i is J-i_i-measurable, we have that 

E[Yi-Yi_i\Fi_i] = {miZi_iWi^i-i + m2{l- Zi_i)W2,i-i) 

> a ■ {Zi-iWi^i-i + (1 — Zi-i) W2^i-i ), 

where the last inequality is because mi, m2 > a. We recall that the variables 
and W2^i-i can only take the values 0 and 1 , and by construction we 
have that Wi^i-i + kF2,i-i > 1 for any f > 1 ; then, we can give a further 
lower bound 


(4.1) E[Yi-Yi_i\Fi-i] > a • (min{Zi_i; 1 - Zi_i}). 

Finally, the result follows by noting that 


min{Zi_i; 1 - Zj_i} 


min{Yi^j_i; Y'2,i-i} ^ minfa,o; ^2,0} 
Yi-i ~ yo + {i -l)b ' 


□ 

Here, we present the lemma on the divergence of the sequences Y^, 
and N2^n- This result is obtained by using the conditional Borel-Cantelli 
lemma. 


Lemma 4 . 2 . Consider the urn model presented in Section 2 . Then, 

/ \ d.s. 

(a) Yn ^ 00; 

(b) min{W,n;A^2,n} 00. 
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Proof. We begin with the proof of part (a). First, notice that Yn = 
Sr=i ~ ^*-i) + Vo- Then, by Theorem 1 in [9], it is sufficient to show 
that 

CX> 

W e ri : [Yj - Yi_i\Fi-i] = oo 

i=l 

occurs with probability one. To this end, we will now use the lower bound 
of Lemma 4.1, so obtaining 


Y,E[Yi-Y,_i\F,.i] 

i=l 


> a 



min{?/i,o;^2,o} 
yo + {i-l)b 


oo. 


Hence, we have that Yn°^' oo. 

We now report the proof of part (b). We will show that W,n oo, since 
the proof for A^ 2 ,n is analogous. Since W,n = SILi by Theorem 1 in [9], 
it is sufficient to show that 


u 




i=l 



occurs with probability one. Then, we obtain 


Y,P{Xi\Fi.i) 

i=l 


i=l 


1/1,0 

^^yo + {i-l)h 


oo. 


Hence, we have that W,n oo. □ 

The following lemma corresponds to Theorem 2.1 of [3], and it is needed 
in the proof of Theorem 2.1. This result provides multiple equivalent ways 
to show the almost sure convergence of a real-valued process. We consider a 
general real-valued process {Zn;n > 0} and two real numbers d (down) and 
u (up), with d < u. The result requires two sequences of times tj{d,u) and 
Tj{d,u) defined as follows: for each j > 0, tj{d,u) represents the time of the 
first up-cross of u after Tj-i(d, u), and Tj{d, u) represents the time of the first 
down-cross of d after tj. Note that tj{d,u) and Tj{d,u) are stopping times, 
since the events {tj{d,u) = k} and {Tj{d,u) = k} depend on {Zn;n < k}, 
which are measurable with respect to Fk. We omit the proof since it is 
reported in Theorem 2.1 of [3], using the same notation. 
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Lemma 4.3. Let {Z^, n >0} be a real-valued process in [0,1]. Let r_i(d, u) 
— 1 and define for every j >0 two stopping times 
(4.2) 


tj {d, u) 


inf{n > Tj-i{d, u) : Zn > u} if {n > Tj{d, u) : Zn > u} 0; 
+00 otherwise. 


Tj{d, u) 


inf{re > tj{d,u) : Zn < d} 
+00 


if {n > tj-i{d,u) : Zn < d} $; 
otherwise. 


Then, the following three events are a.s. equivalent 

(a) Zn converges a.s.; 

(b) for any 0 < d < u < 1, 


lim P {tj{d,u) < oo) = 0; 

j->-oo 

(c) for any 0 < d < u < 1, 

P u) = oo\tj{d, u) < oo) = oo; 

i>i 


using the convention that P {tj^i{d,u) = oo\tj{d,u) < oo) = 1 when 
P {tj{d, u) = oo) = 1. 

The following lemma provides lower bounds for the total number of balls 
in the urn at the times of up-crossings 1)^. The lemma gets used in the proof 
of Theorem 2.1, where conditioning to a fixed number of up-crossing ensures 
to have at least a number of balls Yn determined by the lower bounds of this 
lemma. This result has been taken by Lemma 2.1 of [3]. We omit the proof 
since adaptive thresholds does not play any role during up-crossings and 
the proof reported in Lemma 2.1 of [3] carries over to our model, with Dn 
replaced by Yn. 


Lemma 4.4. For any 0 < d < u < 1, we have that 


(d,u) — 


/ u{l-d) \ 
\d{l-u)J 


— — 


( u{i-d) Y 

\d{l-u)) 


^to(d,u)- 


The following lemma provides a uniform bound for the generalized Polya 
urn with same reinforcement means, which is needed in the proof of The¬ 
orem 2.1. This result has been taken from Lemma 3.2 of [3]. The proof is 
omitted since it is reported in [3]. 
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Lemma 4.5. Consider a generalized Polya urn with mi = m 2 . I/Yq > 2b, 
then 



for every h > 0. 

Here, we provide the proof of Theorem 2.1. 

Proof of Theorem 2.1. Wlog assume mi > m 2 , which implies pn = 
pi^ri and p = pi- We divide the proof in two steps: 

(a) P{ limsup^^oo^n = Pi ) = 1, 

(b) P(lim„^oo^n exists) = 1. 

Proof of part (a): 

We begin by proving that P(limsup^^g^ < pi) = 1. To this end, we 
show that there cannot exist e > 0 and p' > pi such that 



(4.3) 


We prove this by contradiction nsing a comparison argument with a RRU 
model. The proof involves last exit time arguments. Now, suppose (4.3) holds 
and let Ai := {limsnp^^go > p'^}. Let 



and denote the last time the process {pi^n'^n > 1} is above {p'l + pi) /2 by 



Since pi^n Pi, then we have that P 



exists G N snch that 


(4.4) 



Setting Bi := and using (4.4), it follows that 



e < P{Ai) < e/2 + P (Hi n PJ). 
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Now, we show that P {Ai n Bl) = 0. Setting 

Cl = I a; e : lim inf Zn < ^ 1 , 

I n—>oo 2 J 

we decompose P {Ai n Bf) as follows: 

P{Alr^Bl) < P{Ei) + P{E2), 

where Ei = Ai Ci Bf Ci Ci and E 2 = Ai Ci Bf D Cf. 

Consider the term P {E 2 ). Note that on the set Cf, we have 

|liminf„^oo Zn > | and on the set Bf we have {pi^n < i 

n > rie- Hence, since n Cf D E 2 , on the set E 2 we have that = 
l{2n</3i,n} 0- Then, letting tw '■= sup{A: > 1 : Wi^k = 1} we have 

P{E 2 n {tw < 00}) = P{E 2 ) and, on the set £^2, for any n > tw the ARRU 
model can be written as follows: 

{ hl,n+l = Zl^rw 

Y2,n+1 = Y2,r^ + T.tr^ (1 ” £ 3 ,*, 

where = 0 for any i > tw, and H2,i-i = 1 because lR2,i-i + lT2,i-i > 

1 by construction. Now, consider an RRU model {£/^; i > 1} with initial 
composition (T/o,T 2 ^o) ~ (Ti^rv^i ^2,7.^^) a.s.; the reinforcements are dehned 
as = 0 and D^i = D2^Tw+i i Y 1 a.s.; the drawing process is 

modeled by := l[uf^<zE} ~ ^rw+i where {Un]n > 1} is 

the sequence such that = l{[/„<z„} for any n > 1. Formally, this RRU 

model can be described for any n > 1 as follows: 


r Yii+i 

II 

0 ^ 

U.TW 





1 T2«n+1 

II 

0 ^ 

+ Effo (1 - 

zn 

+ E 

n-\-TW( 1 
i=Tw 

X^) D2,i. 

Hence, on 

the set £2 we have 

that 






(Tl,n,U2,n) 

— (Y^ ) 

\ l,n—Tw ’ 2,n—r\Y^ 

a.s., 


for any n 

IV 

Since from [18] P(hmsup„^oo 

z!^ = 

= 0 ) = 1 , on 

the set £2 


we have that {lim sup^^g^ = 0}. This is incompatible with the set A\ 
which includes £ 2 - Hence P (£ 2 ) = 0. 

We now turn to the proof that P (£ 1 ) = 0. To this end, let 
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and note that, since by Lemma 4.2 oo, P{Ci H {r^ < oo}) = P{Ci). 

Moreover, on the set we have that {pi^n < } for any n > rie- We now 

show by induction that on the set n Ci we have {Zn < p'l Vn > r^}- By 
definition we have Zt^ < , and by Lemma 3.1 this implies ^r^+i < p[', 

now, consider an arbitrary n > rp, if Z^ < then by Lemma 3.1 we have 

Zn+i < p'^] if < Zn < p'l we have Wi^n = 0 and so Zn+i < Zn < p'l- 

Hence, since n Ci C Lii, on the set Ei we have {Zn < p'l Vn > Tg}. This 
is incompatible with the set Ai which also includes Ei. Hence P (Ei) = 0. 

Combining all together we have e < e/2 + P (Lii)+ P (£^ 2 ) = e/2, which 
is impossible. Thus, we conclude that P(^f) = P(limsup„^oo < Pi) = 1- 


We now prove that P(limsup„^oo Zn > pi) = 1- To this end, we now show 
that there cannot exist e > 0 and p' < pi such that 


(4.5) P I lim sup Zn < I > e > 0. 

\ n^oo J 

We prove this by contradiction, using a comparison argument with a RRU 
model. Now suppose (4.5) holds and let A 2 := {limsupn^oo < P'i\- 
Let 

t>0 : 

and define the last time the process {pi^n] n > 1} is less than [p'l + pi) /2 by 


’^p'l +P1 

2 


sup{P2} if R 2 / 0; 
0 otherwise. 


Since pi^n Pi, then 
exists Ue € N such that 


we have that P 


f'^ p'l+pi 


< 00 


1. Hence, there 


(4.6) 


P 



< 


e 

2 ' 


Setting B 2 


'l^ p'i+pi 



and using (4.6), it follows that 


e < P{A 2 ) < e/2 + P (H 2 n P^). 


Let £3 := A 2 n Pf- We now show that P (£ 3 )= 0 . On the set A 2 , we have 
{liminfn^oo Zn < p'l} and on the set P|, we have {yOi,n > } for any n > 
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Tie- Hence, on the set we have that Wi^n = ^{Zn<pi n} Then, letting 

riy := sup{A: > 1 : Wi^n = 0} we have P{E3 n {tw < oo}) = P{E^). Now, 
analogously to the proof of P {E2) = 0, we can use comparison arguments 
with the RRU model to show that on the set E^ we have {lim sup^^^o Zn = 
1}. This is incompatible with the set A 2 , which also includes E^. Hence 

P (Es) = 0 . 

Combining all together we have e < e/2 + P {E 3 ) = e/2, which is 

impossible. Thus, we conclude that the event A 2 = {limsup^^^o > Pi} 
occurs with probability one. 


Proof of part (6): 

In part (a), we have shown that P (limsup^^^o = pi) = 1. Therefore, 
if the process {Z„;n > 1 } converges almost surely, then its limit has to be 
equal to pi. First, let d, u, 7 and p[ {d < u < j < p'l < pi) be four constants 
in (0,1). Let {rj(d,u);j > 1} and {tj{d,u)]j > 1} be the sequences of 
random variables defined in (4.2). Since d and u are fixed in this proof, we 
sometimes denote Tj{d,u) by tj and tj{d,u) by tj. It is easy to see that 
and tn are stopping times with respect to {Pn',n > 1 }. 

Recall that, by Lemma 4.3, we have that for every 0 < d < u < 1 

Zn converges a.s. 77 P {tn{d,u) < 00 ) —)• 0, 

OO 

77 P {tn+l{d, u) = Oo\tn{d, u) < Oo) = OO. 

n=l 

Now, to prove that Zn converges a.s., it is sufficient to show that 

P {tn{d, u) < 00 ) —>• 0, 

for all 0 < d < u < 1. Suppose Zn does not converges a.s.. This implies that 
P {tn < 00 ) 4 , (/>i > 0, since P {tn < 00 ) is a non-increasing sequence. We 
will show that for large j there exists a constant (f < 1 dependent on cfi, 
such that 

(4.7) P (tj+i < oo|tj < 00 ) < (j). 

This result implies that YlnE(tn+i = oo|tn < 00 ) = 00 , establishing by 
Lemma 4.3 that P (tn < 00 ) —)• 0, which is a contradiction. 


Consider the term P (tj+i < oo|tj < 00 ). First, let us denote by r^/ the last 
time the process pi^n is below p'^, i.e. 


"■p'l 


supjn > 1 : pi^n < p'li 
0 


if {n > 1 : pi^n < p'll / 0; 
otherwise. 
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Since pi^n Pi, we have that P \ Tpf^ < ooj = 1. Hence, for any e G (O, 
there exists G N such that 

(4.8) jP (v, > n,) < e. 

By denoting Pi (•) = P {-{ti < oo) and using U < Ti < U+i we obtain 
P {ti+l < Oo\ti <Oo) < Pi {Ti < oo) . 


Hence 


(4.9) Pi (ji <oo) < Pi < oo} n {Tp^ < + Pi . 

We start with the second term in (4.9). Note that 


Pi (Vi > ^ 


Vi > 


P {ti < oo) 


< 


Vi > 


4>i 


< e, 


where the last inequality follows from (4.8). 

Now, consider the first term in (4.9). Since the probability is conditioned 
to the set {U < oo}, in what follows we will consider the urn process at 
times n after the stopping time ti. Since we want to show (4.7) for large i, 
we can choose an integer i > rif: and 


i ^ logu(l —d) 

d{l — u) 




so that 

(i) ti> i > a.s.; 

(ii) from Lemma 4.4, we have that Yr^ > 6 / (7 — u) a.s. 

These two properties imply respectively that, on the set {n > ti} 

(i) Pi,n > Pi, since from {r^/^ < n^} we have that n > 

(ii) Zt^ G (u, 7 ), since Zf^-i < u and Zt^ > u and from Lemma 3.1 we have 
that \Zn - Zn-i\ < (7 - u)- 

Now, let us define two sequences of stopping times {tn',n > 1} and 
{Tnin- > 1 }) where f* represents the first time after the process Zt^+n 
up-crosses p}, while t* represents the first time after t* the process Zt^+n 
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down-crosses 7 . Formally, let Tq = 0 and define for every j > 1 two stopping 
times 


(4.10) 



inf{n > : Zn+n > p[} 

+00 


inf{n > t* : Z^+n < 7 } 
-|-oo 


if {n > T* : Zn+n > pU / 0; 
otherwise. 

if {n > t*_-^ : Z^+n < 7 } / 0; 
otherwise. 


Note that, since Zt-j^^-^-i > 7 and ^u+t* < 1-, from (ii) we have that 
Zti+r; e («,7)- 


For any j >0, let {Zn;n> 1} be a RRU model defined as follows: 

(1) (^/,o>^ 2 ,o) = (^i, 7 +r;,hh, 7 +r; 2 T^) a.s., which implies that Z^ = 

u-\-d . 

2 ’ 

( 2 ) the drawing process is modeled by where U^+i = 

Ut^+r*+n+l a.s. and Un is such that Xn = l{c/„<z„_i}; 

(3) the reinforcements are defined as D 2 n+i ~ D 2 ^ti+T*+n+i + (?^i — ^^ 2 )) 

Dln+I = ^i,ti+r*+n+i a.s.; this means E[DlJ = E[DlJ for any 
n > 1; 

(4) the urn process evolves as a RRU model, i.e. for any n > 0 


Kn+l=Kn + K+lDin+l, 


^2,n+l — ^2,n + 






2 ,n+l’ 


y^+1 = Yi 

yJ 


n+l + 7,11+11 


7+1 = 


y3 

^n+1 


We will compare the process {Zf n > 1} with the ARRU process {Zt^+n', n > 
1}. Note that at time n, we have defined only the processes Z^ such that 
r*<n. 


We will prove, by induction, that on the set {r^/^ < n^}, for any j G N and 
for any n < — t* 


(4.11) 


Zi<Z, 


ti+r*+ni 


^2,71 — y 2 ,ti+T* +ni 


Kn < U 


ti+^l+ 11 - 


In other words, we will show, provided that ti > Tp /^, that for each j > 1 the 
process Zn is always dominated by the original process as long as 
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Zt^J^^-*+n is dominated by p'^ (i.e. for n < — r*). By construction we 

have that 

-^0 “ 2 ^ ^ ^ u + T *, ^ 1,0 = 

which immediately implies Y^q > Y 2 ^ti+T*- To this end, we assume (4.11) by 
induction hypothesis. First, we will show that > T 2 ,ti+T*+n+i- Since 

from (4.11) Zn < Zt^+T*+n for n < construction we obtain that 

^n+l — - ^{Un+r*+n<\+T*+n} ~ ^U+T*+n+l- 

As a consequence, since Wn < 1 for any n > 1, we have that 

{Y2,ti+T*+n+l - Y2,ti+T*+r^ ^ ~ ^h+r*+n+l) T)2,q+r*+n+llT2,t.+r*+n 

<{l-K+l)Din+l 

~ ('^ 2 ,n+l ~ ^ 2 ,'n) ) 


which using hypothesis (4.11) implies l 2 „_|_i > T 2 ,ti+T*+n+i- Similarly, we 
now show that Iq „_|_i < Yi^ti+T*+n+i- We have 


Y 


l,ti+T*+n+l - Yl,ti+T*+n 


= X, 


ti+T*+n+lDl,ti+T*+n+l^'^'l,ti+T*+n■ 


iWi 


From (i) we have that, as long as Z remains below p[, Z is also above the 
process p\^n- Since we consider the behavior of when it is below p'^^ 

i.e. n < ~ have that Wi^ti+T*+n = 1- Thus, 


Tl,ti+r*+n+l Yi^ti+T*+n] > ^l,n) > 


which using hypothesis (4.11) implies T/jj+i < Yl^t^^■,-*+n+l■ Thus, we have 
shown that, on the set {r^/ < n^}, for any n < < Zt^+r*+n+i, 

Tgn+i < Yi^ti+rJ+n+i and Yl^^^ > y 2 ,q+r;+n+i hold. 


Now, for any j > 1, let Tj be the stopping time for Zn to exit from (d, u), 


I.e.: 




inf{i?3} if As / 0; 
+00 otherwise. 
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where := {n > 1 : Zn < d or Zn > u}. Note that, on the set 


{Ti < 00 } = <^ inf {Zt^+n} < d 

n>l 


Hence, 




inf 

n>l 




< d 


C {u-o{7; <oo}}. 


Pi (^{Ti < 00 } n {Tp' < nj) < Pi ({u^o {Tj < 00 }} n {Tp' < nj) 


- < 00 } ^ {Vi ^ 

j=0 


n, 


Consider a single term of the series; by setting h = we get 


{{Tj < 00 } n {xp/^ < nj) 




< 


sup > h n {Xp/ < nj 


n>l 


< Pi[snv\Zi-Zl\>h 


n>l 


Note that {Zn,n > 1} is the proportion of red balls in a RRU model with 
same reinforcement means. Then, using Lemma 4.5 we obtain 


Pi sup - Z 5 I > h = Ei 

yn>l 


< Ei 


P 

' b 




s^p\Zi - Zl\>h 

n>l 


4 2 






where Ei [•] = E[-\ti < 00 ]. Moreover, using Lemma 4.4, the right hand side 
can be expressed as 


Ei 


lYu 


p'l(l-7)^V±+2 


.7(i-Pi)y 

Since from Lemma 4.2 Yn converges a.s. to inhnity, and since Xj —)• 00 a.s. 
because Xj > z, we have that Ei tends to zero as i increases. As a 

consequence, we can choose an integer i large enough such that 


Ei 


Y,. 


V/l2 hjY-p'l/^J 


< 2 ’ 


which setting (p = 1/2 + e implies (4.7), i.e. 

P(ti+i < oo|tj < 00 ) < (j) < 1. 
This concludes the proof. 


□ 
















CLT FOR AN ADAPTIVE URN MODEL 


35 


Proof of Corollary 2.1. This corollary has been proved in Proposi¬ 
tion 2.1 of [13] for the MRRU. That proof is only based on the fact that 
the urn proportion converges a.s. to a value within the interval (0,1), 
while the reinforcement rules do not play any role. Hence, the proof used 
in [13] can be applied to the ARRU, since Zn°^' p ^ (0,1) for ARRU using 
Theorem 2.1. □ 


4.2. Proof of the central limit theorem. Before the proof of Theorem 2.2, 
we recall that {Tj;j > 1} is the sequence defined in (3.8) as follows: 



k>l : fj^k G [-b, 0] I 


if { A:> 1 : G [-6,0] } / 0; 

otherwise. 


Fix v G (0,1/2) and, for any j > 1, let rj := 2 and TZj := {tj > rj}. 

The following theorem is critical to the proof of Theorem 2.2. 


Theorem 4.1. Let pi^n and p 2 ,n be as in (2.5). Then, under assump¬ 
tion (2.1) and (2.7), we have that 

(4.12) P(7^J,^.o.) = 0. 

We delay the proof of this theorem to Subsection 4.2.1. 


Proof of Theorem 2.2. Wlog assume mi > m 2 , which implies p = pi. 
To prove the main result, we establish 

(a) ^ Af (0,pi (1 - pi)), and 

(b) “4- 0. 

Finally, result (2.8) is obtained by using Slutsky’s Theorem to combine (a) 
and (b) together. 


Proof of part (a): Let us define a random variable Jni ■= {Xi — E [Aj|4_i]), 

for any n, i G N with i <n. Then, for each n G N, the sequence 

|<S'nj = •^ni\ 1 < j < is a martingale. Now we apply the Martingale 

CLT (MCLT). First note that JA < 1/n for any n G N and \ Jni\ < e for any 
n > e“^; thus 


n ['^ ^]+l 


[e-] + 1 


^ 0 . 


n 
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Also, 


E [ J2. \Fi.i ] = - -EliXni-E [Xni\F.i]f I 

n L 

_ Zj-i (1 — Zj-i) _ 

n ’ 


since pi^n —^ Pi, from Theorem 2.1 we get pi, which implies 

^ I T"- 1 ] = “ ^i-l) a.s^. 

i=l 


n 


pi(i-pi)- 


From MCLT [14], it follows that 
1 


^ (X, - f; [x,|j-,_i]) = 

Vn \ n 

4aA(0,pi(1-pi)). 

We now turn to the proof of part (6). We hrst express 


I ry 

= \ ^i-\ 
n 


V n 


where 


=iPi4-i 

n 


[<?'= 


n—1 


~ /— P^,i 

— “f“ ^271') 


B 


In • — 


E 

i=0 

n —1 


B- 


2 n ■ — 


n 


^ ^ Pl,i) , 

and we recall kn is defined in (2.6) as kn ■= [logg{n)]. We begin with i?i„. 
Note that 

["? "] fcn — 1 <^1 kn — l 

= E E {^»>+. - = EE(-^-.*)> 

j=i i=i 


i=0 


j=l i=l 


where we recall that dj = and Aj^i = pi gj — Zgj^^ for any j > 1 


and 1 < i < dj. Hence 


\Bln\ = 


n 


fcn-l <^3 

EE(-^^<) 

j=l i=l 


kn 1 


< 


1 ^ 
-• V 

/n ^ 




i=i 


/ 


1 
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similarly 


\B2n\ < 


( Y^i=l \ n — 


Now, for any j > 1 define 
(4.13) 

it follows that 


b, : = 


E£i|Afc„,^ 

y/dj 


|-®ln| 4“ |.S2n| ^ 

Now, we have 

|.^ln| 4“ |.S2n| ^ 


1 

j=i 


^ kn/2—1 ^ kn 

E + 7^' ^ 

^ i=l j=kr,/2 


< 




n 


Hin + sup {bi} ■ H 2 r 
\i>kr,/2 I 


where 


fcn/2-l 


Hin := 


^ E V^’ F2„:=^Ev^- 

4=1 4=1 


Using dj = (<7 — l)g-^ we express 

. -- fc„/2-l 

=^. (^) 

4=1 


j _ 




n 


n 


4=1 


j _ 




n 


n 


T^-iy ’ 

Vg- 1 
79-1 


Since n > it follows that Hin < C and H 2 n < y^C, where C = (^ 
Thus, 


+ |H2n| E 


/ sup,^{6J \ ^ ^ \ ^ 

\ <rn J \i>fc„/2 / 


79-7 ■ 


To conclude the proof we will show that bj *14' 0. 
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a + v 


First, fix an arbitrary constant v € (0,1/2) and let Vj := 2 for any j > 1; 

then, write 


bj = 


1 


i=i 




1 


Vv^ i=i 

= Fij + F2j, 

Let us consider term Fij, we have that 




1 


j=r,+l 


Fij = 






qH 


I n 

J i=l 


V 9 -I V rj ^ 


Ei^, 




2=1 


since dj = {q — l)q^ and rj/\/^ = g-’z. Now, for any i = l,..,rj we note 
that 

— Zqj\ + 

hence, we have 

Fij < Eij + £'2j + E^j, 


where 


£^ij 


E- 


q^ 


2 j 


E- 


V?- 1 


Fj 

-Ei2. 

i=l 


gz+i ■ 


3j 


■ |Pl,g2-i “ Pl,g2 I- 


V?- 1 

Let us consider the term Ey. Since from Lemma 3.1 we have {Z^. — Z^-il < 
b/Yk _i, we have that 


Eij < 


br^ 


V?- 1 VV9 - 1 




Then, by using Markov’s inequality we obtain 


E^- 

i=i 


3^(1+'") 




>e < 




^ 00 

/E® 


i=i 




gz J 


q-jih-) < 

i=i 


< 00 
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where C = supi^^i{E[k/Yk]} is finite from Theorem 3.1. Thus, from the 
Borel-Cantelli lemma it follows that Eij 0. 

Now, consider the term E 2 j. We have 


P ( lim Uj>k{E 2 j > e} ) 

yfc—)-oo y 


<p{ 

lim VJj>hlZn 


\k^OQ 

/ / 

+ p\ 

lim U,>fc < 


1 fc—>oo 1 




V?- 1 


• \^j,dj\ > e > n7^5 


where the term P (lim^^oo Uj>fc7^j) = 0 from Theorem 4.1. Then, by using 
Markov’s inequality we obtain 


i=i 


(1+1) f 


-1 


> e> n7^^ I < M, 


where 


1 °° 

M ;= -^E 


1=1 


;(l + l)f 


V?- 1 




Now, for any j > 1 let us introduce the set Qj := {rj > dj}. Using C 
from Vj < dj, and by multiplying and dividing by we have that 


M = 


< 














. g-(l+i)(i-f) 
. g-(l+i)(i-f) 


< 


1 


sup \ E 


{E[,‘+‘|AtA|l2c 


}E 


j = l 


ey/q - 1 \fc>i 
< oo, 

using Theorem 3.2 and the result follows from the Borel-Cantelli lemma. 

Let us consider the term E^j. For any e > 0, by using Markov’s inequality 
we have 

1 


P {Esj > e) < 


Wq -1 


E 


■ \pl,gi - Pl,qi-A 
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The right-hand side (RHS) of the above expression can be rewritten as 

(f 




E 


- 1 

Now, by decomposing the last expectation into 


E 


92 • \h,qi - Pi,qj-A 


= E 


■ \Pl,ql - Pl,qi-^ 
pectation into 
qi ■ \pi - pi^gj-i 


E 


q2 ■ \pi 


(jJ 


we can see that 

00 

J2P{E3j>e) < 

i=i 


’SUPfc>i {t; • \pl - Pl^qA } 

Wq -1 


i=i 


which is finite because of (2.7). Hence, by another application the Borel- 
Cantelli lemma, Hsj Qj then, we have Fij 0 . 

Finally, let us consider term . First, we multiply and divide by {dj — rj) q 
to obtain F 2 j = cjF^j, where 


a = 


dj - fj 


^ ' 

QWdj 


E3j = 


di — r. 


E d|A 


J,*l- 


^ ^ i=r j+1 


Since Cj -A y/q — 1, let us focus on F^j. Since P {JZj,i.o.) = 0 (Theorem 4.1), 
it is sufficient to show that F^jX^c por any e > 0, by Markov’s 

inequality it follows that 


p{{Fy>,]nnf) < E ^ 9’IAy,|iKC 


1 


e \ d-j — r,- , 

J J 2 =r,+l 


Now, since 


E E\p\^jMnc\ < (._max |F;[g5|Aj-*|l^cj|) 

- - \ 2 - F A “ 1 “ - d . y j 

-r> ._L1 \ J J / 


we have that 

P 


({Fsj > e} n TZf) <-{ max | j 
^ e \i=rj+l,..,dj I 

- ( max 

e \i=rj+l,..,dj I 

1 


< 


[e 


\ 


3 

J 



} 


_ 3 _ 

q 2 


sup < max \E\q’^\Aki\lT,c \ \ i 
e \k>l\i=[rk] + F;dk^ L ' 


_ 3 _ 

q 2 
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where the last inequality follows from Theorem 3.2. Now, summing over j 
we have that 


^P{{F 3 j>e}mzf) < C^q-i < oo. 
i=i i=i 

Now, using the Borel-Cantelli lemma we get that F 2 j 0, which concludes 
the proof. □ 

4.2.1. Proof of Theorem 4-T 

Proof. Wlog assume mi > m 2 , which implies pn = pi,n and p = pi. To 
prove (4.12) we need to study the sequence of sets {TZj ;j > 1 }. On the set 
TZj, the urn proportion does not cross the thresholds at times , ..,q^ + rj. 
Hence, TZj will be included in AjUBj, where Aj and Bj represent the events 
in which the urn proportion is always above and below, respectively, the 
thresholds at times q^,..,q^ + O- show that Aj and Bj cannot occur 
i.o., we need to appropriately express them by using the following scaling 
processes: 

(a) fj^k = Ygjj^kAj^k = Ygj+k {Pi,qj - Zqj+k), defined for any j > 1 and 
any k = l,..,dj. This process models the closeness among the urn 
proportion and the adaptive threshold. 

(b) Tn = Yn{pi — Zn), dehned for any n > 1. This process models the 
closeness among the urn proportion and the limit of the threshold’s 
sequence. 

(c) := Ygj^^ [pi - Pi gi), defined for any j > 1 and k = l, ..,dj. This 
process models the closeness between the adaptive threshold and its 
limit. 

Let us now dehne formally the sets Aj and Bj. First, note that if the urn 
proportion crosses the threshold at time {q^ + k), then Tgj+fc • < 0, 

since only one among Yqj+k and is within the interval [—6,0]. Thus, 

from the dehnition of Tj in (3.8), we have that 

{Aj^k-i ■ ^j,k < 0} C {Tj < k} . 

This implies that 

Y {Aj^k-i • ^j,k > 0 }} 

= { {^j,k < 0} } U { f^k=i {^j,k > 0}} • 
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Since - T-^k\ 


we can write 


TZj C j^j u Bj , 


where 


^k=l^hki 


B, := 


I 'k=l^j,k^ 


and Pj-fc := for k = 1, ..,rj. 

The idea to prove that these events cannot occur infinitely often is the 
following: consider Aj (for instance) and rewrite the set as follows: 
(4.14) 




qO+Tj 


< T, 


(pi)\ _ 


Arj J 



q^+i 




(Pl) 

Arj 


-T„j 


where the last inequality follows using telescopic series. In the set 
we have a sum of bounded random variables, i.e. (Tg^+j — , whose 

means are strictly positive on Aj, because Aj in included in '^j,k] hence, 

provided that the difference — T^,) increases with j slower than rj, 

we could prove that the set cannot occur infinitely often. Roughly speaking, 
it means that, if the adaptive threshold pi ^j is not far enough from the 
urn proportion ^qj 1 then the average increments of the urn proportion make 
very likely that crosses pi ^j before + rj. Similar arguments apply 

for Bj. More formally, fix e > 0 and define the set Cj as follows: 



so that Cf is the set where the difference — T„j I increases with j slower 

J J "I 3 ^ 

than rj. Hence, it follows that 


Uj C {Aj-Cj] U {Bj-Cj] U Cj, 


and the result (4.12) is obtained by showing that 


P {Aj — Cj,i.o.) = P {Bj — Cj,i.o.) = P {Cj,i.o.) = 0. 

We will now begin with the proof of P {Aj — Cj,i.o.) = P {Bj — Cj,i.o.) = 
0. From (4.14) we note that, on the set Cp, 




. 2 = 1 
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As a consequence, we have 

~ — I ^k=i ^j,k n fj I , 

B,-C, c }■ 

Now, consider the increments for i = I, ..jrj contained in 

the sets £j and Tj above; recall that 

l) Pi ^ql+i) ^2,ql +i^^2,ql +i—l 

(1 Pi) ^ql +i^l,ql +i^^l,ql +i—l- 

Fix an arbitrarily small ei > 0 and introduce two collections of i.i.d. random 
variables (Ai,..,A,.^) and [Bi,Br^) defined as follows: 

■= Pi (^1 ~ ^{U^j+i<Pl+ci}) ^2,qi+ii 

■— Pi ~ ^{Ugj+i<Pl-ei}) ^2,ql+i ~ ~ Pi) _^_^<pi-e}^l,qi +iJ 

where (l7gi+i, ••, t^qi+i) are the i.i.d. (0,1) uniform random variables such 
that := l{u J ■ 

First note that, by construction, on the set Aj we have > Pi^gj+fc}, 

and hence Aj C = 0}. Thus, since using (2.7) we have 

Zn Pi by Theorem 2.1, on the set Aj we have that 

{{Tgj+i - Tgj^i_i) >Ai}, iel, ..,rj, 

occurs with probability 1 as n —)• oo. Similarly, by construction, on the set 
Bj we have {Zgj+k < Pi,qi+k}, and hence Bj C nl^=^{Wi^gj+k = !}• 
Thus, using Zn pi, on the set Aj we have that the event 

{{Tgj+i - Tgj^i_i) <Bi}, iel, ..,rj, 


occurs with probability 1 as n —)■ oo. As a consequence, for large j we have 
that 

P{Aj -Cj,i-o.) < P i'^Ai < efqiA-o\ 


,i=l 
' D 


P {Bj - CjA-o.) < P J]] Si > 


-efq 2 ,i-o. 


\ 2 = 1 
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Set 




i=l 


i=l 


We will now use Cliernoff’s upper bounds on the i.i.d. bounded random 
variables Ai and Bi (see (3.7)). First notice that 

(1) E [Ai] = pi{l- pi- e) 7712 > 0, 

(2) E [Bi] = Pi (1 - Pi + e) 7772 - (1 - pi) (pi - e) 7711 < 0, 

(3) \Ai\,\Bi\ < b a.s. for any i > 1. 

Note that PAj can be written as P {Sj < Cj ■ Fl[S'j]), where Sj = YllLii^i/b) 
and 

•2 ^ 

_ ej 

“ rjE[A^]/b' 

since Cj —>• 0 , we can define an integer jg such that cj < cg for any j > jg, 
so that 

P [Sj < Cj • E[Sj]) < P [Sj < cg • E[Sj ]). 

Hence, by using (3.7), for any j > jg we have that 

Paj < 


which converges to zero exponentially fast since 

E[Sj] = r, 


E [Ai] ■ i+i. 

--i nJ -y 


We can repeat the same arguments for Psj^ with the i.i.d. random variables 
{—Bi + b)/2b e (0,1) for i = in this case, Cj tends to a constant 

c < 1, so that the proof follows with cg G (c, 1). Thus, 


yielding 


^^{PAj + Pbj) < oo, 

1=1 

P {Aj — Cj,i-o-) = P {Bj — Cj,i-o ) = 0. 


We will now show that P {Cj,i-0-) = 0. Note that since 


and 


Pqi — Pqi{Pl “ Pl,g 7 -l) + Yqi{Pl,qi-i “ Z^j) — + Pj-l,dj-i^ 
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it follows that 




which implies that 


{Cj,i.o.} C U {\fj^dj \ > . 


Now, since < ho + bn, it follows that 

{Cj,i.o.} C {gij,i.o.} U {02j,ho.}, 


where 


Gij ■= 


G2j : = 


ho i, » I -2 ^ 


We will now show that P{Qij,i.o.) = 0. By using the Markov’s inequality 
we have 


OO Q 7 CXD / 


En 


i=i 


^ V 


+ 1 


£1 


- Pi,qA 


3qb (Yq 


e \bq 


\ °° 1 

+‘)<^e4 


i=i 


where 


C := supi;| q'^Api -Pi,qA I 
k>l L J J 


< OO, 


< OO 


from (2.7). Hence, using the Borel-Cantelli lemma, it follows that 
P{gij,i.o.) = 0. 

Now, consider ^ 2 j. Let Pj := {j~^qi • | > e} and since 


P{G2jP-o.) = P {Pj,i.O.) 

we now focus on Pj. First, for each j > 1, we recall that Qj = {tj > dj} 
and we decompose Pj as follows: 

Pj c Q, u {P, n Qf }, 


which leads to 

P{Pj,i.o.) < P{Qj,i.o.) + P {Pj ,i.o) . 
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First, consider P (Pj n , i.o^ . By using Markov’s inequality we have 

OO OO 

^p{n,naf) < 


J =1 


J =1 


_1 
q 2 


J ej^ 


< 


SUPfc>l j-E g*’ • |Afc,dJlgC I) 


OO _1 
j2 ’ 

j=l ■' 


which is finite from Theorem 3.2. Hence, again from the Borel-Cantelli 
lemma we have that 

P {Pj n Q^, i.o) = 0. 

We will now show that P {Qj,i.o.) = 0. To this end, we can follow the 
same arguments used in the first part of this proof, except that here we 
define 

In this case, to show P {Cj,i.o.) = 0 we have to prove that the following two 
events cannot occur infinitely often 

(i) Gsj ■= I + l) \Pi - Pl,qA > ’ 

(n) Q4j := I 1/91 - Zgj \ > . 

Result (i) is implied by (2.7), while (ii) follows from Theorem 2.1. Hence, 
we have that 

P {Cj,i.o.) = 0. 


Then, similarly to the first part of the proof, we deal with the sets Aj — Cj 
and Bj — Cj by applying Chernoff’s upper bound to the probabilities 

/ 

( J2Ai<eq^ 


and Psj = P I Bi > —eq^ 


i=l 


which implies PAj < oo and X^^i Pb] < oo. Hence, from the Borel- 

Cantelli lemma we get 


P {Aj — Cj,i.o.) = P{Bj—Cj) = 0. 
which implies P {Qj,i.o.) = 0. This concludes the proof. 


□ 


Remark 4.1. The result of Theorem 4A continues to hold if (2.7) is 
not satisfied, but (2.2) and condition (cl) hold. Moreover, since in the proof 
we use Theorem 3.1, if (2.7) does not hold condition (c2) must he assumed 
(see Remark 3.1). 
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4.3. Proof of Proposition 2.1. 

Proof. Wlog assume mi > m 2 , which implies pn = pi,n and p = pi. 
First, we have 

(4.15) E [n\pi^n - Pil"^] = -E 

n 


i=0 


and note that 

n—l 


Pi) 

i=0 


kn 

l,qi+i Pi) +2<n} 

kn 1 


{Pl,ql ~ Pi) + 

j=0 


n — q 


{h. 


Pi) , 


where we recall kn is defined in (2.6) as kn '■= [logq{n)]. Since dj = {q — l)q^, 
the LHS of (4.15) is equal to 

2 n 


(<?-l)^ 


E 


n 


kji 1 

{pi,qi - pi)) + 

j=0 


n — q 


Q 


^ j {Pl,q>=n Pi) 


and, defining Cj := ,/q^Pi^qi — Pil, we can rewrite the last expression as 
follows: 


iQ-^y 


E 


n 


'kn-l 

E + 

j=0 


n — q 


{q - 1 ) 


1 \ 

2 - 

Ckn 



Now, using Cauchy Schwartz inequality and using y ^ y/Q the 

above expectation is less than or equal to 


kn kn 


jl=0j2=0 


Je 


E 

k 1 

V 

L 


L 


Now, by the symmetry in Kn, we can use the following decomposition 


kn k-fi 

y/ kn y/kn 

y/ kn 

kn 

kn 

kn 

EE(-) 

= EE(-) 

+ 2E 

E (•) + 

E 

E 

jl=0j2=0 

Ti 

0 

to 

II 

0 

n=0j. 

2=y/^ 

ji=VPkj: 

3 — y/ kn 


kn y/ kn 

kn 

kn 




s^EEl 

m E 

E 




ii—0j2 —0 j2=\/Ki 
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we obtain 


kn \/ kji 

Kn < sup {E[c)]] ■ 2 ^ (v^) 

ii=oi2=o 

kn kn 


J 2 


i>i 


+ max ^ iVQyHVQy'' 

y/kn _— _ _— 

Jl — ykn j2—\kn 


32 


— H“ ^2n’ 

Now, consider i^in; we have that 


Km < sup {E [c]]} ■ 2 

j>i \ VQ-k 

and by multiplying for {q — l)^/n we obtain 




V9-1 




n 


n 


Using (2.7) we have that supj>i | is finite. Moreover, since n < 

qkn+i |3y definition of kn, we have that 


(Vq) 


^nH“l 


n 






- 1 


0 . 


n 


Similarly, we can consider K 2 n and write 


K 2 n< max {E[c^]] 
Vf^<j<kn 


\v^) 


^n + 1 


- 1 




Then, by multiplying for {q — Vf' jn we obtain 


q-1 


max {E[cJ]}' 


(V?) 


^Tl + 1 


- 1 


n 


.\/9 1 / \/kk<j<kn 

and from (2.7) and n < we have maxy^<^<^^ | is finite and 


(A) 




- 1 


n 


< q- 

















CLT FOR AN ADAPTIVE URN MODEL 


49 


Then, combining all together we obtain 



< g(l + ^/qf ■ limsupT; [n|pi,„ - pi\^] 


which is finite becanse of condition (2.7). 


□ 


4.4. Proof of Corollary 2.2. 

Proof. To prove this result, we apply Theorem 2.2 to the urn model 
with fixed thresholds, i.e. p\^n = P\ and p^^n = P 2 for all n > 0, since in this 


case Pn = P for all n> 0. 


□ 


4.5. Remarks on the CLT for Z^. In this subsection, we discuss the 
second-order behavior of the proportion of balls in the urn in the ARRU 
model. Specifically, we establish CLT of for some specific subsequences 
{nj',j > 1} and we highlight the challenges to the proof of a full CLT. To 
this end, let us assume that 


Vn{Pn - p) Z, 


(4.16) 


where Z is a Gaussian random variable with zero mean and variance cr^ > 0. 


It is worth noticing that (4.16) is usually verified in applications, since pn 
is typically a continuous function of maximum likelihood estimators, e.g. 
see (2.10). 

Wlog, assume mi > m 2 and consider the sequence {y/n{pi — Zn)]n > 1}. 
Now, along the subsequence {q^]j > 1} 


(4.17) y^{pi-Zgj) = \/^{pi - Pi,gj-i) + y^{pi^qj-i - Zqj). 
Using (4.16), it follows that 



As for the second term in (4.17), it can be expressed as 
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where we recall TZj := {tj > rj} and rj := with u G (0,1/2). The 

hrst term converges to zero a.s. from Theorem 4.1, while using rj_i < dj-i 
from Theorem 3.2 we have 




\Aj- 






I—I l{rj_i<dj_i} 


0 . 


Thus, 

V^(pi - Zqj) A- y/q- Z. 

Now, consider the subsequence {q^ + r^; j > 1}. Now, 

(4.18) \/q^{pi Zqj+Tj ) = {Pl - Pl,qi) + V^(/5l 

As before, from (4.16) we have that 


{Pi — Pi,qo) Z. 

Once again expressing the second term in (4.17) below, 

'/^{Pl,qi ~ ^qi+rj) = ^j,rj 

= + Vo^Aj^rA-Rp 

one can show the first term in the RHS converges to zero a.s. from Theo¬ 
rem 4.1. The second term in the RHS tends to zero in from Theorem 3.2. 
Thus, 

\l + 'f'jiPl - ^qi+Tj) 

A crucial result to obtain CLT for {Znp,j > 1}, with rij = q^ and rij = 
q^ + Tj, is Theorem 4.1, which establishes P{Tj > nj,i.o.) = 0. From these 
results, it follows that the asymptotic distribution of Zn only involves times 
t € ^j{Q^ + Fj! 9'^'*'^)) for these times. Theorem 3.2 establishes a uniform 
bound for n\Zn — Pn\- However, at times t G Uj{q^,q^ + Tj) it seems difficult 
to obtain the detailed behavior of {Zn — pn)- This gap needs to be handled 
for a CLT for {Z„;n > 1}. This is beyond the scope of the current paper. 


5. Simulation studies. In this section, we describe some simulation 
studies that illustrate the theoretical results presented in Section 2 in the 
context of clinical trials. We recall from subsection 2.3 that, in the context of 
clinical trials, the random variables and ^ 2 ,n are interpreted as potential 
responses to competing treatments 7i and Ti, whose distributions pi and 
P 2 depend on parameters and 62 respectively. Let 6 = { 61 , 62 ). Now, 
letting /i and /2 are two continuous functions, we recall that pi = fi{6) 
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and p 2 = / 2 (^)- Moreover, the adaptive thresholds pi^n and p 2 ,n are defined 
as follows: 

/5l,n := fl and P2,n ■= /2 (^2,n) , Vn > 1, 

where and 02 ,n are the adaptive estimators of 0i and 02 after the first 
n allocations. 

The main goal of this section is to illustrate the asymptotic behavior of 
the allocation proportion Ni^n/n and of the parameter estimators 0^- Simu¬ 
lations are performed with N = 10^ independent urn processes, each which 
evolve following the model described in Section 2 with adaptive thresh¬ 
olds pi^n and p 2 ,n that change at exponential times {q^',j > 1}, with q = 
1.25, [see (2.5)]. For all the N urn processes we used initial composition 
(yi, 0 )y 2 ,o) = (2,2) and sample size n = 200. The functions fi and /2 are 
chosen as in (2.11) with p = 0.75. We analyze both Bernoulli and Gaussian 
responses. 


5.1. Bernoulli responses. We assume responses to treatments Ti and Ti 
are from Bernoulli distributions with parameters p\ and p 2 , respectively. In 
this case, 0 = {pi,P 2 ) is the vector of unknown parameters. We examine two 
target allocations: 

(a) r]{e) = (1 - pi) l{2-pi- P2), proposed by [22]; 

(b) 7 /( 0 ) = (^ -L y/p^), proposed by [20]. 

Hence, from (2.11) with p = 0.75, we have 

Pi = 0.25 • 1-L 0.75 • 7 /(pi,P 2), and p 2 = 0.25 • 0-F 0.75 •r/(pi,p 2 )- 


In Table 5.1 we report the simulation results on the mean and standard 
error of the allocation proportion Ni^n/n and of the estimators pi^n and 
P2,n, defined as 


Pl,n = 


Nl,n 


and 


P2,n = 


N2,n 


Hence, 


Pgn = 0.25 • 1 -F 0.75 • T/(pi,„,P2,n), 


and 


P2,n = 0.25 • 0 -F 0.75 • 7]{pi^n,P2,n)- 
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Table 1 

Simulation of Ni_n/n and 6n are given for different designs, with mean square 
errors given in parenthesis. The target allocation is pi = {1 — p) ■ 1 + p ■ r]{pi,P 2 ) 
with p = 0.75. Simulation used N = 10® ARRU processes with n = 200 and 
changes at times {q^',j > 1} with q = 1.25. Initial composition (yi,o, 2 / 2 ,o) = (2,2). 


Pi 

P2 

pi 

Ni ^ n/n 

Pl.n 

P2.n 



( a ) p = (1 - 

Pi)/(2 -Pi 

-P 2 ) 

0.9 

0.7 

0.44 

0 . 44 ( 0 . 07 ) 

0 . 89 ( 0 . 03 ) 

0 . 7 ( 0 . 04 ) 

0.9 

0.5 

0.38 

0 . 41 ( 0 . 06 ) 

0 . 89 ( 0 . 03 ) 

0 . 50 ( 0 . 05 ) 

0.9 

0.3 

0.34 

0 . 40 ( 0 . 07 ) 

0 . 89 ( 0 . 03 ) 

0 . 30 ( 0 . 04 ) 

0.9 

0.1 

0.33 

0 . 43 ( 0 . 12 ) 

0 . 89 ( 0 . 03 ) 

0 . 11 ( 0 . 03 ) 

0.7 

0.5 

0.53 

0 . 50 ( 0 . 07 ) 

0 . 70 ( 0 . 05 ) 

0 . 50 ( 0 . 05 ) 

0.7 

0.3 

0.48 

0 . 48 ( 0 . 05 ) 

0 . 70 ( 0 . 05 ) 

0 . 30 ( 0 . 04 ) 

0.7 

0.1 

0.44 

0 . 48 ( 0 . 06 ) 

0 . 70 ( 0 . 05 ) 

0 . 11 ( 0 . 03 ) 

0.5 

0.3 

0.56 

0 . 53 ( 0 . 06 ) 

0 . 50 ( 0 . 05 ) 

0 . 30 ( 0 . 05 ) 

0.5 

0.1 

0.52 

0 . 53 ( 0 . 04 ) 

0 . 50 ( 0 . 05 ) 

0 . 11 ( 0 . 03 ) 

0.3 

0.1 

0.58 

0 . 56 ( 0 . 05 ) 

0 . 30 ( 0 . 04 ) 

0 . 11 ( 0 . 03 ) 


( b ) V = ^/^/(^/^ + ^/ 72 ) 

0.9 

0.7 

0.65 

0 . 57 ( 0 . 11 ) 

0 . 89 ( 0 . 03 ) 

0 . 69 ( 0 . 05 ) 

0.9 

0.5 

0.68 

0 . 63 ( 0 . 08 ) 

0 . 89 ( 0 . 03 ) 

0 . 50 ( 0 . 06 ) 

0.9 

0.3 

0.73 

0 . 69 ( 0 . 06 ) 

0 . 89 ( 0 . 03 ) 

0 . 30 ( 0 . 06 ) 

0.9 

0.1 

0.81 

0 . 76 ( 0 . 07 ) 

0 . 89 ( 0 . 02 ) 

0 . 11 ( 0 . 04 ) 

0.7 

0.5 

0.66 

0 . 58 ( 0 . 11 ) 

0 . 69 ( 0 . 04 ) 

0 . 50 ( 0 . 06 ) 

0.7 

0.3 

0.70 

0 . 66 ( 0 . 07 ) 

0 . 70 ( 0 . 04 ) 

0 . 30 ( 0 . 06 ) 

0.7 

0.1 

0.79 

0 . 74 ( 0 . 07 ) 

0 . 70 ( 0 . 04 ) 

0 . 12 ( 0 . 04 ) 

0.5 

0.3 

0.67 

0 . 60 ( 0 . 10 ) 

0 . 50 ( 0 . 05 ) 

0 . 30 ( 0 . 05 ) 

0.5 

0.1 

0.77 

0 . 70 ( 0 . 08 ) 

0 . 50 ( 0 . 04 ) 

0 . 11 ( 0 . 04 ) 

0.3 

0.1 

0.73 

0 . 64 ( 0 . 11 ) 

0 . 30 ( 0 . 04 ) 

0 . 11 ( 0 . 03 ) 


5.2. Gaussian responses. We now assume responses to treatments 71 and 
72 are from a Gaussian distribution with parameters (mi, af) and (m 2 , ul), 
respectively. In this case, 9 = (mi, uf, m 2 , ul) is the vector of unknown 
parameters. We examine two target allocation: 

(c) r]{9) = ai/ (fJi + 0 - 2 ), used in [14]; 

(d) r/(0) = axy/mf,! (cri,/m^ + a 2 ^ffnt), proposed by [23]. 

Hence, from (2.11) with p = 0.75, we have 

Pi = 0.25 • 1+ 0.75 • r/(0), and p 2 = 0.25 • 0 + 0.75 • p(0). 

In Table 5.2 we report the simulation results on the mean and standard error 
of the allocation proportion Ni^rijn and the parameter estimators u^i^ri and 
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Table 2 

Simulations of Ni^n/n and are given for different designs, with mean square 
errors given in parenthesis. The target allocation is pi = {1 — p) ■ 1 + p ■ r]{0) with 
p = 0.75. Simulation used N = 10® ARRU processes with n = 200 and changes at 
times {q^',j > 1} with q = 1.25. Initial composition (yi,OiJ/ 2 ,o) = (2,2). 


mi 

m2 


0-2 

Pi 

Ni,n/n 

0-^1,n 

0-^2,n 





(c) g-- 

= Oil (cti + CT 2 ) 


10 

5 

1 

1 

0.63 

0.61(0.05) 

1.01(0.13) 

1.01(0.16) 

8 

5 

1 

1 

0.63 

0.59(0.07) 

1.01(0.13) 

1.01(0.16) 

6 

5 

1 

1 

0.63 

0.55(0.12) 

1.01(0.14) 

1.01(0.15) 

10 

5 

4 

1 

0.75 

0.73(0.06) 

4.00(0.47) 

1.01(0.20) 

8 

5 

4 

1 

0.75 

0.71(0.07) 

4.00(0.48) 

1.01(0.19) 

6 

5 

4 

1 

0.75 

0.66(0.13) 

4.03(0.50) 

1.01(0.18) 

10 

5 

1 

4 

0.50 

0.49(0.05) 

1.01(0.14) 

4.00(0.57) 

8 

5 

1 

4 

0.50 

0.48(0.07) 

1.01(0.15) 

4.03(0.56) 

6 

5 

1 

4 

0.50 

0.43(0.11) 

1.01(0.16) 

4.03(0.54) 



(d) g = 

= oi^/mf/{aiyPrn^ + a2^/mf) 


10 

5 

1 

1 

0.56 

0.55(0.05) 

1.01(0.14) 

1.01(0.15) 

8 

5 

1 

1 

0.58 

0.55(0.07) 

1.01(0.14) 

1.01(0.15) 

6 

5 

1 

1 

0.61 

0.53(0.12) 

1.01(0.14) 

1.01(0.15) 

10 

5 

4 

1 

0.69 

0.67(0.06) 

4.03(0.49) 

1.01(0.18) 

8 

5 

4 

1 

0.71 

0.67(0.07) 

4.03(0.49) 

1.01(0.18) 

6 

5 

4 

1 

0.73 

0.65(0.13) 

4.03(0.51) 

1.01(0.18) 

10 

5 

1 

4 

0.45 

0.44(0.05) 

1.01(0.15) 

4.03(0.54) 

8 

5 

1 

4 

0.46 

0.44(0.07) 

1.01(0.16) 

4.03(0.54) 

6 

5 

1 

4 

0.48 

0.42(0.11) 

1.01(0.16) 

4.03(0.53) 


T^ 2 ,n, defined as 


T^Ln = 


Er=i (?i,i - ^ 1 ,' 


iVi 


and 


l,n 


(r‘^2,n = 


Er=l (1 “ ('^2,1 — 1^2,nY 


N, 


2,n 


where rhi^n = EEi ^nd m2,n = Er=i(l “ ^i)^2,i/N2,n- Hence, 

,5i,„ = 0.25-1 + 0.75-7/(4), 


and 

;52,n = 0.25-0 + 0.75-7/(4). 

The results show that our methods target the true parameters effectively. 
In real clinical trials, further calibration may be performed to reduce small 
bias. 
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6. Extensions to mnlti-color urn models. It is important to note 
that all the results presented in this paper can be extended to the case of 
K > 2 colors, when 3j G {1, ..,K} such that rrij > ruk for any k ^ j. In the 
context of clinical trials, the functions fj should be interpreted as the target 
allocations for Nj^n/n when Tj is the superior treatment, and the variables 
Wj^n should be all dehned as ^{z„<p. }■ 
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